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ABSTRACT 



Searchers and scientists have been studying neural networks for many years hoping 
to achieve human-like performance in the fields of speech and pattern recognition and 
classification. In this study, we are first going to make an introduction to the field of 
artificial neural networks, then we are going to describe some of the neural nets used in 
the pattern recognition and classification. A computer simulation program from an al- 
gorithmic approach for each one of these networks will be constructed and used to im- 
plement the operation of the net. Its ability will be demonstrated in differentiating 
between different patterns and even correcting a noisy pattern and recognizing it. The 
Hopfield network, the Hamming network and the Carpenter / Grossberg network will 
be individually utilized in developing an algorithm for pattern recognition and classi- 
fication. 

The maximum-likelihood sequence estimation function will be mapped onto a neural 
network structure. The application of this structure computations for data detection in 
digital communications receivers will be described. A computer simulation program will 
be constructed and used to show that neural networks offer attractive implementation 
alternatives for MLSE. 
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I. INTRODUCTION 



A. WHAT IS A NEURAL NETWORK : 

A neural network is a highly parallel network with many interconnections between 
analog computational elements or nodes. In other words, a neural net is a system com- 
posed of many simple processing elements operating in parallel whose function is deter- 
mined by network structure, connection strengths and the processing performed at 
computing elements or nodes. These nodes offer one possible solution to the problem 
of obtaining the massive parallelism and computational requirements that are presumed 
to be required for such problems as pattern recognition and classification that we are 
going to discuss in this study. Artificial neural nets are of interest primarily because they 
may be able to emulate the speed and performance of real biological neural nets using 
many simple slow computational elements operating in parallel to obtain high compu- 
tation rates. Figure 1 illustrates this definition. 




A small interconnected neural network is presented on the left side of this figure and 
one simple type of processing element or node is presented on the right side [Ref. 1]. This 
particular node forms the sum of N weighted inputs presented on X input links and 
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passes the result through a nonlinearity out on one output link. In addition, the weights 
on the input links can be adapted based on information concerning the correctness of 
the output. Neural nets almost always include an inherent nonlinearity and require 
primarily local connectivity between nodes which are almost always nonlinear, typically 
analog, and may be slow compared to modern digital circuitry. Nodes may also include 
temporal integration and other types of time dependencies and also mathematical oper- 
ations more complex than summation. [Ref. 1] 



Architectures and processing elements used in neural network models are simplified 
versions of those observed in biological nervous systems. Figure 2 illustrates a number 
of different types of biological neurons and a small biological neural network. [Ref. 1] 




Characteristics of biological neural networks that artificial neural network models 
hope to provide include: [Ref. 1] 

• Fault tolerance to loss of a small number of computational elements. 

• Insensitivity to small variations between computational elements. 

• The need for primarily local connectivity and local learning rules. 

• Real time response. 

• Parallelism. 
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Work in neural networks is generally oriented towards achieving rather high-level 
intelligent functions, such as pattern recognition, categorization, and associative mem- 
on - . The biological knowledge of these functions is far from complete, but it is very clear 
that neurons and synapses are the fundamental devices used. It is also clear that these 
devices are not programmed in the conventional manner; rather, problem-specific 
knowledge is acquired by a learning process which alters the neuronal parameters di- 
rectly. These are the two principal facts of biology that have been applied to neural 
networks. They are the equivalent of the transistor and of the logically structured pro- 
gram in conventional computers. In addition, the algorithms for calculating the output 
of a model neuron from its input and the high synaptic connectivity used in model net- 
works both derive from biological observations. [Ref. 1] 

Modern neuroscience provides a great wealth of additional information that has 
only just begun to be applied to neural network modeling. This is because the path from 
this more recent biological information to the desired intelligent functions is relatively 
tenuous, and the simple ideas of neurons, synapses and learning, are themselves sur- 
prisingly powerful. [Ref. 1] 

The few principles of neurons, synapses, and learning constitute the biological 
foundation of most neural networks. They are, of course, insufficient to specify a net- 
work with the kinds of high-level intelligent functions mentioned above. In order to 
achieve these functions, the biological foundation is supplemented with cleverly invented 
ideas, some drawn from other disciplines, notably physics. This non-biological approach 
is appropriate considering the technological goals of the research, the lack of clear al- 
ternative biological solutions, and the possibility that future research will verify that such 
imported ideas are in fact biological. However, if biological realism is not sufficiently 
maintained, neural networks will lose the ability to interact profitably with neuroscience. 
[Ref. 1] 

B. NEURAL NETWORKS IN PATTERN RECOGNITION AND 
CLASSIFICATION : 

Pattern recognition and classification is an area where neural nets have proven to 
be very successful. 

A taxonomy of six important neural nets that can be used for recognition and clas- 
sification of unknown patterns is presented in Figure 3. 
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Figure 3. A taxonomy of Neural Network Classification and Clustering Models 



[Ref. 2] 



This taxonomy is first divided between networks with binary' and continuous valued 
inputs. Below this, nets are divided between those Trained with or without supervision. 
Overall, adaptive neural networks can be trained using three types of training proce- 
dures: [Ref. 1] 

• Supervised training, which requires labeled training data and an external teacher. 
The teacher knows the desired correct response and provides a feedback error sig- 
nal after each trial. This is sometimes called reinforcement learning, or learning with 
a critic when the teacher only indicates whether a response was correct or incorrect 
and does not provide detailed error information. 

• Unsupervised training, sometimes called self-organization , uses unlabeled training 
data and requires no external teacher. Data is presented and internal categories or 
clusters are formed which compress the amount of input data that must be proc- 
essed at higher levels without losing important information. Clustering is an im- 
portant component of many pattern classification procedures. It is sometimes 
called vector quantization when used to convert analog inputs into a binary form 
suitable for transmission or storage. 

• Self-supervised training is used by automata which monitor performance internally 
and require no external teacher. For example, automata which learn to track a 
moving spot by controlling simulated eye muscles can generate an error signal 
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based on the distance between the position of the spot on a simulated retina and 
the center of fovea of the retina. Self-supervision is sometimes called learning-by- 
doing or learning by experimentation. 

Nets trained with supervision such as the Hopfield net and perceptrons are used as 
associative memories or as classifiers. The teacher provides side information or labels 
that specify the correct class for new input patterns during training. Most traditional 
statistical classifiers, such as Gaussian classifiers, are trained with supervision using la- 
beled training data [Ref. 2]. Nets trained without supervision, such as the Kohonen's 
feature-map forming nets [Ref. 2], are used as vector quantizers or to form clusters. The 
teacher does not provide these nets with any information concerning the correct class 
during training. The classical K-means [Ref, 2] and the leader clustering algorithm 
[Ref. 2] are trained without supervision. 

In this study, we are going to focus on the use of neural net classifiers for fixed 
patterns with binary input elements. We are going to implement and simulate, for dif- 
ferent cases of input patterns, the supervised Hopfield net, the Hamming net which is a 
neural net implementation of the optimum classifier for binary inputs, and the unsuper- 
vised leader clustering algorithm of the Carpenter / Grossberg net. 

C. NEURAL NETWORKS AS MLSE RECEIVERS OF BINARY SIGNALS IN 
GAUSSIAN NOISE : 

In this study, we are also going to focus on the use of neural network based 
maximum-likelihood sequence estimation (MLSE) receiver structure. In particular, the 
problem of detecting digital data symbols transmitted over a time-dispersive time- 
varying channel in the presence of additive Gaussian noise will be considered . We are 
going to computer implement this neural network structure and simulate it for stationary 
or time-varying transmission channel. Results of these simulations will be provided to 
show that neural networks offer attractive implementation alternatives for MLSE 
[Ref. 3]. 
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II. THE HOPFIELD NETWORK 



A. GENERALITIES: 

In recent years, an upsurging interest in neural networks made of highly parallel 
computational elements connected in patterns that are reminiscent of biological neural 
nets has caught attention of researchers and scientists. In particular, more recent work 
has explored the ability of a neural model described by Hopfield to serve as a content- 
addressable memory (classifier). This network retrieves one of the M stored exemplars 
given an input pattern which is a noisy version of one of these exemplars. A classifier 
determines which of the M exemplar patterns is most similar to the noisy input pattern. 

In the following study, we will focus on the classification problem because a 
content-addressable memory is essentially a classifier which outputs the exemplar for the 
selected class instead of an index to the class. Classification is a fundamental operation 
that is essential to the important problem of speech and image recognition whether 
achieved by biological or artificial means. 

Past studies have demonstrated that the Hopfield model can be used as a content- 
addressable memory for random input patterns and to classify binary patterns created 
from radar cross sections, consonants and vowels extracted from spoken words, and lines 
in an image. These results demonstrate that a neural network based on the Hopfield 
model can perform classification. In addition, Hopfield models have been successfully 
applied to other problems, such as, the traveling salesman problem, the Analog to Dig- 
ital (A-D) converter problem, and the signal decomposition problem. [Ref. 4] 

B. OPERATION OF THE HOPFIELD NETWORK: 

The unit used by a number of scientists is the familiar binary' threshold unit (Me 

X 

Culloch-Pitts neuron) whose output is 1 if and only if Y, w u s i> where 0 <j< M — 1; 

<=i 

otherwise 0, where X is the number of elements or bits in a pattern, s, is the current value 
of the ith input and w v is the corresponding synaptic weight from i to unit j whose 
threshold is 0 r In the McCulloch-Pitts networks, every neuron processes its inputs to 
determine a new output at each time step [Ref. 5], By contrast, a Hopfield net is a net- 
work of such units subject to the updating rule: "Pick a unit at random. If the sum of 
the weights on connections to other active units is positive, turn it on. Otherwise turn 
it off'. The operation of this network is described as we first apply input values of an 
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unknown pattern at time zero through the bottom threshold-logic nodes. This forces the 
output of the net to match the unknown pattern at time zero: 

Hi(0) = x, 0 < i < N — 1 (2-1) 

where n,{t) is the output of node i at time t and x , is element i of the input pattern taking 
on the values +1 or -1. Following this initialization, the network iterates in discrete 
time steps using the given equation: 



Vj{t + 1 )=f h 



-.v-i 



o<;< m - 1 



L ('=0 



( 2 - 2 ) 



In this equation f is a modified hard-limiter function and is the weight applied 
to the output of node i that feeds to node j. Previousely, we have assumed the elements 
of the input vector x take on values + 1 and —1, respectively, for the + 1 and —1 states, 
then f h is the symmetric hard-limiting function. 



AM = 




, if a > 0 
, if a < 0 



(2-3) 



J 

+ 1 


r>] 

L 






Ce 

-i 





Figure 4. The hard-limiting function used in the Hopfield network 



The weights are set using exemplar patterns for all stored classes. 
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(2-4) 




0 i =j ,0 < ij < N - 1 



*i*j l *J 



where x? is element i of the exemplar for pattern s. The output of each node is fed to 
every other node with a weight that is symmetric, and each node does not feedback to 
itself. After convergence, the output of the net is the final pattern represented by the 
outputs of the nodes. 



The network is considered to have converged when the outputs no longer change 
on successive iterations. When the Hopfield net is used as an associative memory, the 
network output after convergence is used directly as the restored memory. When used 
as a classifier, the output of the Hopfield net after convergence must be compared to the 
M exemplars to determine if it matches an exemplar exactly. If it does, the output is the 
exemplar that best matches the output pattern. If it does not, then a "no match" result 
occurs. 

Hopfield first demonstrated that when the net is trained with M exemplar patterns 
using Equation 2-4, and an exemplar is presented at time zero, then the final pattern in 
the net after convergence will be one of the M exemplars with a high probability if. 



The exemplars thus form stable states of the net. Hopfield's statistical results were 
obtained with randomly generated exemplars. It is possible and relatively easy to select 
a set of M exemplars that satisfies Equation 2-6, but does not form stable states in the 
Hopfield net. These exemplars must have many elements in common. When an exemplar 
for one of these patterns is presented at time zero, the network does not converge to any 
of the trained exemplars. Instead, it converges to a spurious pattern never seen before. 
This problem of spurious states also occurs when a noisy exemplar is presented to the 
net. Even when the M exemplars are stable states of the net, there is no guarantee that 
noisy versions of these exemplars passed through discrete, memoryless channels and 
presented at time zero will converge to the original exemplars. Hopfield, for example, 
observed that the number of spurious states found increases substantially as more ele- 
ments in the input exemplar are corrupted. 



oo) i = 0,\,...,N — 1 



(2-5) 



M < 0.1 5 A’ 



(2-6) 
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The Hopfield neural network can be used as a classifier only when: 

1. The exemplars for the patterns to be classified form stable states and converge to 
themselves when presented at time zero as input. 

2. A mechanism is provided to determine which of the M exemplars the net is closest 
to after convergence. 

The first requirement is a necessary condition for a proper classification operation. 
The second is necessary because the Hopfield net by itself is not a neural-net classifier, 
but is more like a preprocessor which still requires a classification net to select which of 
the M classes an output pattern is closest to . 

OUTPUTS IVelid After Convergence] 




INPUTS (Applied at time Zero) 



Figure 5. The Hopfield net used as a content-addressble memory [Ref. 2] 

It is difficult to satisfy the requirement that exemplars form stable states without 
actually running the Hopfield net. In general, patterns that are more random will satisfy 
this requirement more easily than patterns with many bits in common. 

C. IMPLEMENTATION OF THE HOPFIELD NETWORK: 

The Hopfield network can be used either as an associative memory or as a content- 
addressable memory which is described in this study. The Hopfield net shown in Figure 
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5, has N nodes containing hard-limiting nonlinearities and binary inputs and outputs 
taking on the values + 1 and —1. The output of each node is fed back to all other nodes 
via weights denoted vvv. 

A computer algorithm to implement the operation of this net as a content- 
addressable memory can be summarized in four necessary' steps : [Ref. 2] 

Step 1. Assign Connection weights : using Equation 2-4 

Step 2. Initialize with Unknown Input Pattern : using Equation 2-1 

Step 3. Iterate Until Convergence : using Equation 2-2, the process is repeated until 
node outputs remain unchanged with further iterations. The node outputs then represent 
the exemplar pattern that best matches the unknown input. 

Step 4. Repeat by Going to Step 2 

The weights are first set using Equation 2-4 and elements of the M stored exemplar 
patterns as the operation algorithm of the net stated in the first step. Eight patterns 
(M = 8), shown in Figure 6, have been selected to simulate this algorithm and were 
stored in the memory of the network. 

For convenience, these eight patterns were selected to be 120 nodes (12 by 10 ma- 
trices) each. The only limitation in the choice of N (number of nodes) is the time that 
the net will take to iterate and converge to an output pattern or respond with a "no 
match". After assigning connection weights, an unknown input pattern is imposed on the 
net at time zero by forcing the output of the net to match the unknown pattern. Then, 
the net iterates in discrete time steps. The net is considered to have converged when 
outputs no longer change with further iterations. The pattern specified by the node 
outputs, after convergence is reached, is the net output. [Ref. 4] 

Using the Hopfield net as a classifier, the output will be compared to every one of 
the \1 class patterns. If the output matches an exemplar, the classification is terminated 
and the output is that class whose exemplar best matches the output pattern. If it does 
not, then a "no match" result occurs. 
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Figure 6. The Eight stored patterns 
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D. SIMULATION OF THE HOPFIELD NETWORK: 

Using the Fortran program provided in Appendix A, we simulated the operation of 
the Hopfield net when used as an addressable-content memory (classifier) for different 
inputs. For convenience and simplicity, the M class patterns were first introduced to the 
net as a matrix of N by M (120 by 8), where each vector colunm of 120 nodes represents 
the 12 by 10 matrix representation of the class. The first vector colunm represents the 
pattern of a 'O', the second of a T, the third of a '2', the fourth of a '3', the fifth of a '4', 
the sixth of a '6', the seventh of the block pattern representation of the 'point' and the 
eight and last of a '9'. The elements of each class pattern take on the values of + 1 for 
a ' black pixel ' and —1 for a ' . 

The behavior of the network is simulated first by presenting the pattern of digit '3' 
as an input pattern. To make it more interesting, a corrupted version of this pattern is 
achieved by randomly reversing each bit, of the matrix representation of digit 3, inde- 
pendently from + 1 to — 1 and vice versa with a probability of 0.25. Implementing this 
pattern is equivalent to receiving noise corrupted bits of a digit in a noisy communi- 
cation channel. 

The corrupted input pattern was then imposed on the net at time zero. After the 
first iteration, the net still can not tell which class the input pattern corresponds to. As 
more iteratic ns took place, the output becomes more and more like the correct exemplar 
pattern of the digit 3, as you can see in this simulation result provided in Figure 7. 

Then, at the third iteration, the net has converged and the output, as can be seen, 
is indeed the pattern of the digit 3. Only three iterations were sufficient for the net to 
converge to the corrected digit and to recognize it as a 3. Now, we tried to see if we 
present a corrupted pattern of another digit, how many iterations will be used to con- 
verge to the correct result? Will it take only three iterations to do so ? The input pattern 
that we used was of digit '9' and using the same procedure we randomly reversed each 
bit from +1 to —1 and vice versa with the same probability and error distribution as 
before. The response of the network to this input pattern is illustrated in Figures 8 and 
9. 
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THE UNKNOWN INPUT PATTERN TO THE HOPFIELD NETWORK, ON THE LEFT AS IM- 
POSED ON THE NETWORK AND ON THE RIGHT IN A MUCH CLEAR REPRESENTATION 



WHERE EVERY ( ■ 


) 


REPLACES 


A 1 


AND 


EVERY 


( 


• 


) REPLACES 


A 


-1: 






-1 -1 -1 1 


1 


1 


1 


-1 


-1 


1 








■ 


■ 


■ 


■ 






■ 


-1 1-1 1 


1 


1 


1 


1 


1 


1 




■ 


















1 -1 -1 -1 


1 


-1 


-1 


1 


-1 


-1 


■ 








■ 






■ 


. 




1 -1 -1 -1 


-1 


-1 


-1 


-1 


1 


-1 


■ 
















■ 




-1 -1 -1 -1 


-1 


1 


-1 


-1 


1 


-1 












■ 






■ 




-1 -1 1 -1 


1 


1 


1 


1 


-1 


-1 


. 




■ 




■ 


■ 


■ 


■ 


# 




1 -1 -1 -1 


1 


1 


-1 


-1 


-1 


-1 


■ 








■ 


■ 


, 




. 




-1 -1 -1 1 


-1 


-1 


-1 


-1 


1 


1 


. 






■ 


. 






. 


■ 


■ 


-1 -1 -1 -1 


-1 


-1 


1 


1 


1 


-1 














■ 


■ 


■ 




-1 -1 -1 1 


-1 


-1 


-1 


1 


-1 


-1 








■ 




, 


. 


■ 






1111 


1 


-1 


1 


-1 


-1 


-1 


■ 


■ 


■ 


■ 


■ 


. 


■ 




. 




-1 -1 -1 1 


1 


1 


-1 


1 


-1 


1 


• 


* 


• 


■ 


■ 


■ 


• 


■ 


• 


■ 


THE OUTPUT OF THE 


HOPFIELD 


NETWORK 


LOOKS 


LIKE 


THE 


FOLLOWING FOR 


THE 



UNKNOWN INPUT PATTERN PRESENTED. THE PATTERN ON THE LEFT CORRESPONDS 
TO THE OUTPUT AFTER THE 1ST ITERATION WHILE THE PATTERN IN THE MIDDLE 
CORRESPONDS TO THE OUTPUT AFTER THE 2ND ITERATION, THE PATTERN ON THE 
RIGHT CORRESPONDS TO THE OUTPUT AFTER THE 3RD ITERATION. 






CLASSIFICATION OF THE UNKNOWN INPUT PATTERN: 



AT THIS POINT, FURTHER ITERATIONS WON'T MAKE ANY CHANGE ON THE OUTPUT 
OF THE NETWORK AND THE PATTERN SPECIFIED BY THE OUTPUT NODES IS THE 
NET'S OUTPUT. THE TASK OF THE NET NOW IS TO CLASSIFY THE INPUT AS AN 
ALREADY KNOWN PATTERN OR A NO MATCH WILL OCCUR. 

AFTER CLASSIFICATION, THE OUTPUT PATTERN OF THE HOPFIELD NET MATCHES 
BEST THE PATTERN OF DIGIT THREE. 



Figure 7. Hopfielf net response to the first input pattern 
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THE OUTPUT OF THE HOPFIELD NETWORK LOOKS LIKE THE FOLLOWING FOR THE 
UNKNOWN INPUT PATTERN PRESENTED. THE PATTERN ON THE LEFT CORRESPONDS 
TO THE OUTPUT AFTER THE 1ST ITERATION WHILE THE PATTERN ON THE RIGHT 
CORRESPONDS TO THE OUTPUT AFTER THE 2ND ITERATION. 



THE OUTPUT OF THE HOPFIELD NETWORK AFTER THE 3RD ITERATION IS PRESE- 
NTED ON THE LEFT WHILE THE PATTERN ON THE RIGHT CORRESPONDS TO THE 
OUTPUT AFTER THE 4TH ITERATION. 




Figure 8. Hopfield net response to the second input pattern 
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CLASSIFICATION OF THE UNKNOWN INPUT PATTERN: 



AT THIS POINT, FURTHER ITERATIONS WON'T MAKE ANY CHANGE ON THE OUTPUT 
OF THE NETWORK AND THE PATTERN SPECIFIED BY THE OUTPUT NODES IS THE 
NET'S OUTPUT. THE TASK OF THE NET NOW IS TO CLASSIFY THE INPUT AS AN 
ALREADY KNOWN PATTERN OR A NO MATCH WILL OCCUR. 

AFTER CLASSIFICATION, THE OUTPUT PATTERN OF THE HOPFIELD NET MATCHES 
BEST THE PATTERN OF DIGIT NINE. 



Figure 9. Classification response of the Hopfield net to the second input pattern 

Then, using the same probability of error and distribution of errors in the input 
pattern as for digit '3', we conclude from the result of this simulation that the number 
of iterations needed to get the right answer depends on how close the input pattern was 
to a stored one. This may explain why there is a difference in the number of iterations 
taken by the net to converge to the right answer. We may say that the corrupted input 
pattern of digit '3' was closer to the perfect exemplar for the '3' than the noise disturbed 
input pattern of digit '9' is to the perfect '9' using the same noise corrupted bits distrib- 
ution and the same probability of error as for digit '3'. 

Using the noise disturbed pattern for digit '9' as input to the net, but now with dif- 
ferent error distribution and the same probability of error (0.25), the simulation and re- 
sults were as shown in Figure 10. 

The result shows perfectly that the number of iterations that the network needs to 
converge depends on the error distribution in the input pattern and not on the proba- 
bility of error. 

Now as a conclusion after all these simulations and results, one could say that when 
imposing a perfect input pattern on the net, the network will take only one iteration to 
recognize it as a stored one. To verify this finding, we take the pattern of digit '2' and 
present it as the input pattern to the net at time zero without making any change in its 
elements (a perfect exemplar of digit '2')- The response of the net to this perfect input 
pattern is presented in Figure 11. 
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THE UNKNOWN INPUT PATTERN TO THE HOPFIELD NETWORK, ON THE LEFT AS IM- 
POSED ON THE NETWORK AND ON THE RIGHT IN A MUCH CLEAR REPRESENTATION 
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THE OUPUT OF THE HOPFIELD NETWORK LOOKS LIKE THE FOLLOWING FOR THE 
UNKNOWN INPUT PATTERN PRESENTED. THE PATTERN ON THE LEFT CORRESPONDS 
TO THE OUTPUT AFTER THE 1ST ITERATION WHILE THE PATTERN IN THE MIDDLE 
CORRESPONDS TO THE OUTPUT AFTER THE 2ND ITERATION, THE PATTERN ON THE 
RIGHT CORRESPONDS TO THE OUTPUT AFTER THE 3RD ITERATION. 







CLASSIFICATION OF THE UNKNOWN INPUT PATTERN: 



AT THIS POINT, FURTHER ITERATIONS WON'T MAKE ANY CHANGE ON THE OUTPUT 
OF THE NETWORK AND THE PATTERN SPECIFIED BY THE OUTPUT NODES IS THE 
NET'S OUTPUT. THE TASK OF THE NET NOW IS TO CLASSIFY THE INPUT AS AN 
ALREADY KNOWN PATTERN OR A NO MATCH WILL OCCUR. 

AFTER CLASSIFICATION, THE OUTPUT PATTERN OF THE HOPFIELD NET MATCHES 
BEST THE PATTERN OF DIGIT NINE. 



Figure 10. Hopfield net response to the third input pattern 
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THE UNKNOWN INPUT PATTERN TO THE HOPFIELD NETWORK, ON THE LEFT AS IM- 
POSED ON THE NETWORK AND ON THE RIGHT IN A MUCH CLEAR REPRESENTATION 
WHERE EVERY ( « ) REPLACES A 1 AND EVERY ( . ) REPLACES A -1: 
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THE OUTPUT OF THE HOPFIELD NETWORK LOOKS LIKE THE FOLLOWING FOR THE 
UNKNOWN INPUT PATTERN PRESENTED. THE PATTERN SHOWN HERE IS THE NET'S 
OUTPUT AFTER THE 1ST ITERATION. 



CLASSIFICATION OF THE UNKNOWN INPUT PATTERN: 



AT THIS POINT, FURTHER ITERATIONS WON'T MAKE ANY CHANGE ON THE OUTPUT 
OF THE NETWORK AND THE PATTERN SPECIFIED BY THE OUTPUT NODES IS THE 
NET'S OUTPUT. THE TASK OF THE NET NOW IS TO CLASSIFY THE INPUT AS AN 
ALREADY KNOWN PATTERN OR A NO MATCH WILL OCCUR. 

AFTER CLASSIFICATION, THE OUTPUT PATTERN OF THE HOPFIELD NET MATCHES 
BEST THE PATTERN OF DIGIT TWO. 



Figure 11. Hopfield net response to the perfect input pattern 
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The result was as we thought; it took the net only one iteration to recognize the 
input pattern as one of the M class stored patterns. From these simulations we can 
conclude that the Hopfield network when used as a classifier can be useful in a com- 
munication receiver where its task is to recognize the received bits, also to correct the 
corrupted ones and to recognize them. However, the Hopfield net is only iterating be- 
tween an input pattern and the ones that are already stored in the memory of the net. 
The number of these patterns (\1) is a limitation to the proper operation of the net as 
a classifier because of the convergence condition demonstrated by Hopfield, which states 
that the net will converge with high probability if M < 0.15:Y. A non-learning network 
is what the Hopfield net is, compared to other networks that we are going to describe 
in the following chapters. However, it has an important advantage over the others, its 
ability to recognize patterns even in noisy environement as long as the original pattern 
was stored in its memory prior to its use, otherwise, a "no match" will occur. 
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III. THE HAMMING NETWORK 



A. GENERALITIES : 

The Hopfield net, as we have seen, is often tested on problems of pattern recognition 
and classification by taking an input exemplar and reversing its bits randomly with a 
certain probability. The classifier in this study will calculate the Hamming distance to 
the exemplar of each class and select that class with the minimum Hamming distance to 
the specified input pattern. The Hamming distance is the number of bits in the input 
which do not match the corresponding exemplar bits. A net, which will be called the 
Hamming net, implements this algorithm using neural net component. Instead of cal- 
culating the Hamming distance directly, we will calculate N minus the Hamming dis- 
tance and maximize this function, where N is the number of elements or bits in a pattern 
representation. [Ref 4] 

X minus the Hamming distance can be calculated from a weighted sum of the X 
elements of the input vector. If the elements of the input pattern to the net take on the 
values + 1 and —1 for the respective states, then 



A' -At 



ham 



= C; + 



A — 1 

I 

f=0 



w ij*i 



(3 - 1) 



where, 




(3-2) 



and 



9 = 



N_ 

2 



(3-3) 



Here xj is the value of element i of the exemplar for class j. When all elements in the 
input vector match an exemplar exactly, each element in the sum of Equation 3-1 adds 
and adding c j given in Equation 3-3 gives a total of X. Whenever an element in the 
input pattern does not match the corresponding element in the exemplar, the prior total 
is decremented by 1 as required. [Ref. 4] 
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On the other hand, when elements of the input pattern x take on the values 0 and 
+ 1 for the —1 and + 1 states, respectively, N minus the Hamming distance can be cal- 
culated from : 



where, 



and 



N-i\i 



ham 




(3-4) 




if xj = +1 

if 4 = 0 



(3-5) 




(3-6) 



In the above equation, A' represents the number of zero elements in the exemplar 
for class j. When all elements in the input pattern match an exemplar exactly, the sum 
in Equation 3-4 adds up to the number of positive input elements. This is added to the 
number of zero input elements results in N, as desired. The sum is reduced by one 
whenever a zero input element that matches an exemplar becomes positive, and when- 
ever a positive input element that matches an exemplar becomes zero. [Ref. 4] 

Here we have made a brief introduction to the Hamming net used as a classifier, as 
the net that calculates the Hamming distance to exemplars for all classes and then select 
that class which produces the minimum Hamming distance to the input pattern. Also , 
we have introduced two kinds of input patterns, one which elements take on + 1 and 
— 1 values, the other + 1 and 0 values, for the + 1 and —1 states, respectively. Its con- 
sequences on the calculation of N minus the Hamming distance were also introduced. 
The next paragraph will discuss the Hamming net in further detail and illustrate how the 
selection of the minimum is made. 

B. OPERATION OF THE HAMMING NETWORK : 

Two neural nets that are logically required to implement an optimum classifier for 
binary patterns will be assembled to form the Hamming net. One net forms the weighted 
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sum to calculate quantities related to the likelihood of the different classes. The second 
net picks the maximum. [Ref. 4] 

The first net that forms weighted sums is presented in Figure 12. An input pattern 
x is applied at the bottom of this net and an output pattern y is produced at the top. 




Figure 12. Feed-forward neural net used to calculate M weighted sums from the N 
elements of the input pattern [Ref. 4] 



The first layer of nodes sends values of the input pattern to the links feeding the 
second layer. The second layer of nodes uses nonlinear threshold logic elements to sum 
weighted values of the inputs and add internal offsets [Ref 4]. Output values from the 
second layer are 




+ 




0 <j<M - 1 



(3-7) 



where, 
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(3-8) 



m = 



j”a ifa>0 
1 0 //a < 0 



In these equations, /(a) is a nonlinear function that models the nonlinearity inherent 
in a biological neuron, c, is an internal offset associated with each threshold logic node, 
and vv, y are positive or negative weights associated with the links [Ref. 4]. 

A number of different nets can be used to pick the maximum value from the y s out- 
puts of the feed-forward neural net shown in Figure 12. In situations where it is only 
important to know if the input matches a stored state very closely, it is sufficient to 
identify those second-level nodes in Figure 12, with output values that exceed a specified 
threshold. This can be performed by modifying the constant c, added in Equation 3-7 
such that only the output of those nodes corresponding to closely matching stored states 
are positive. [Ref. 4] 



outputs 



' M-2 



M-1 




Figure 13. The iterative neural net called "maxnet" that picks the maximum of M 
inputs [Ref. 4] 
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In the more general situation, a net must select the maximum over the My) values. 
There are many different neural net structures which perform this task. A less complex 
net that uses feedback connections to pick the maximum output y t ( referred to as a 
niaxnet ) is presented in Figure 13. [Ref. 4] 

Although this net is similar in structure to the Hopfield net (Figure 5), it uses 
threshold-logic nodes, relative to the threshold-logic nonlinear function described in 
Equation 3-8, rather than hard-limiting nodes and feeds the output of each node back 
to its input instead of disallowing this feedback path. The maxnet is a fully connected 
net made up of only M threshold logic nodes with internal thresholds set to zero. Input 
values are applied at time zero through the input nodes on the bottom of Figure 13. This 
initializes node outputs for each node at time zero [ju,(0)] to the input values : 

Hj( 0) = yj j = 0,1,..., A/ - 2 , M - 1 (3-9) 

The network then iterates to find the maximum using the following equation : 



+ 1 ) = / 



Ah-w - £ y Wo 



i*j 



0 < i , j < M — 1 



(3 - 10) 



In this equation, / is the threshold logic function described in Equation 3-8. Each 
node inhibits all other nodes with a value equal to the node's output multiplied by a 
small negative weight (e) which is less than -yy Each node also feeds back to itself with 
unity gain. After convergence, only that output node corresponding to the maximum 
input will have a nonzero value. This value will generally be less than the original time 
zero value of that node. The output values of the net are thus simply the node output 
values after convergence : 



zj = Hj(o o) j = 0,1 ,..., M - 2,M - 1 



(3-11) 



The maxnet will converge and find the maximum input when 






(3-12) 



By convergence, we mean that the output nodes stop changing in time and only the 
output of one node corresponding to the maximum input is positive. Applying the 
threshold logic function / on each one of these output nodes will result in only one 
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nonzero output node, the previous positive one, and a zero value for all the others. The 
nonzero node corresponds to that exemplar class which best matches the input pattern. 




The block diagram of the complete Hamming net, when used as a classifier, is 
completed by putting together the feed-forward neural net referred to as the lower sub- 
net (calculates the weighted sums) , and the maxnet referred to as the upper subnet (se- 
lects the node with the maximum output value). The complete Hamming net is then as 
shown in Figure 14. 

C. IMPLEMENTATION OF THE HAMMING NET : 

The operation algorithm of the Hamming net as a classifier can be described in four 
steps which the net must follow' to classify a certain input pattern. The four steps of the 
algorithm are : [Ref. 2] 
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Step 1. Assign connection Weights and Offsets 
In the lower subnet : 



2 



0</^ A r — 1 , 0 <j<, M - 1 



(3-13) 



and 



Sj = -y 0 <j < A/ — 1 



(3 - 14) 



In the upper subnet : 




+ 1, k = 



— c, /:¥=/, c < y , 0 < k,l < M — 1 



(3-15) 



In these equations, w if is the connection weight from input i to node j in the lower 
subnet and d is the threshold in that node. The connection weight from node k to node 
1 in the upper subnet is t kl and all thresholds in this subnet are zero, x{ is element i of 
exemplar j . 

Step 2. Initialize with Unknown Input Pattern 



In this equation, ju/r) is the output of node j in the upper subnet at time t, jc, is ele- 
ment i of the input, and f, is the threshold logic nonlinearity. It is assumed that the 
maximum input to this nonlinearity never causes the output to saturate. 

Step 3. Iterate Until Convergence : using Equation 3-10, this process is repeated until 
convergence occurs after which the output of only one node remains positive. 

Step 4. Repeat by Going to Step 2 

First, weights and thresholds are set using the N elements of each one of the \1 
stored patterns, as shown in Step 1 of the above algorithm. Then a binary input pattern 




(3-16) 
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with X bits (X = 120 elements in this implementation) is presented at the bottom of the 
Hamming net at time zero. The X bits of the input pattern were chosen to take on the 
values + 1 and —1 for + 1 and —1 states, respectively. This input pattern must be pre- 
sented long enough to allow the lower subnet of the net to calculate matching scores 
which are going to be fed to the upper subnet (maxnet) allowing it to settle and initialize 
its outputs. These matching scores are equal to X minus the Hamming distances be- 
tween the input and each one of the M exemplar patterns. This operation is done by 
using the equation given in Step 2 of the previous algorithm. The input is then removed 
and the maxnet iterates until convergence using the iteration formula 3-10. By conver- 
gence, we mean that the output of only one node, corresponding to one of the M stored 
patterns, is a nonzero value. Classification is then terminated and the nonzero maxnet 
output node will point out the selected class that best matches the input pattern. 

The M stored patterns used in the implementation of the Hamming net when used 
as a classifier, were choosen to be similar to those used in the previous chapter; however, 
here we are using 10 exemplar patterns (M = 10) instead of 8 used earlier. The 10 stored 
patterns (Figures 15 and 16) consist of 120 nodes (X= 120) each as in the previous 
chapter (12 by 10 representation matrices). These 10 classes were choosen to represent 
all digits. 




Figure 15. The first four stored patterns 
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Figure 16. The rest of the 10 stored patterns 
D. SIMULATION OF THE HAMMING NETWORK: 



Using the Fortran program provided in Appendix B. we simulated the operation of 
the Hamming net when it is used as a classifier. The input pattern is choosen to be the 
digit '3' pattern which in the same fashion as for the Hopfield net, reversing its bits 
randomly from +1 to —1 and vice versa with the same probability (0.25). 

The behavior of the Hamming net is illustrated in the output of the simulation 
program provided in Figure 17. 
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THE UNKNOWN INPUT PATTERN TO THE HAMMING NETWORK, ON THE LEFT AS IM- 
POSED ON THE NETWORK AND ON THE RIGHT IN A MUCH CLEAR REPRESENTATION 
WHERE EVERY ( • ) REPLACES A 1 AND EVERY ( . ) REPLACES A -1: 
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OUTPUT OF 


THE 


HAMMING 


NETWORK, 
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OUTPUT NODE VALUES FOR THE CORRESPONDING CLASSES AT A CERTAIN NUMBER 
OF ITERATIONS: 
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CLASSIFICATION OF THE UNKNOWN INPUT PATTERN: 

THEN, THE DISTURBED UNKNOWN INPUT PATTERN TO THE HAMMING NETWORK 
CORRESPONDS TO THE PATTERN STORED OF THE CLASS THREE. 



Figure 17. Response of the Hamming net to the first input pattern 
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For more clarity, we are going to illustrate the behavior of the Hamming net in the 
following plots. Each plot corresponds to the output of the net at the corresponding time 
step from time zero until its convergence. 




Figure 18. The output of the Hamming net at t = 1 for digit "3" 
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Figure 19. The output of the Hamming net at t = 2 for digit "3" 
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Figure 20. The output of the Hamming net at t = 3 for digit "3" 
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Figure 21. The output of the Hamming net at t = 4 for digit "3" 



32 





Figure 22. The output of the Hamming net at t = 5 for digit "3" 
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Figure 23. The output of the Hamming net at t = 6 for digit "3" 
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Figure 24. The output of the Hamming net at t = 7 for digit "3" 
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Figure 25. The output of the Hamming net at t = 8 for digit " 3 " 

As you can see, the net has effectively converged to the correct class corresponding 
to digit '3' and only S iterations were required to do so. By convergence, we mean that 
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the output nodes stop changing in time steps, and only the output of one node is a 
nonzero value. In our case, it was the output node of class 3. We say that the net has 
converged to the class that best matches the input pattern which is given by the position 
of the nonzero output node within the others. Further iterations will not change the 
output node values. 

The proof of convergence depends primarily on the fact the inhibition to the node 
containing the maximum value, in our case it was the node corresponding to class '3', 
is always less than the inhibition to other nodes. This explains the fact that all output 
node values were decreasing on successive time steps. The output node values of those 
classes that are close to the input pattern were decreasing in time, but not in the same 
fashion as the classes that are totally different from the input pattern which were de- 
creasing faster. At convergence, the inhibition to the node with the maximum value re- 
duces to zero. [Ref. 4] 



where, inhibit) is the second term of the right hand side of Equation 3-10, ^,(0 is the 
output of node i at time t and t 0 is the inhibition weight between nodes in the upper 
subnet and it's given by the following formula, 



Node 3 corresponds to the maximum input, then on the first iteration, the inhibition 
to this node was less than the inhibition to all other nodes. This follows because all node 
outputs are positive and the sum of all outputs, excluding one in Equation 3-17, will be 
minimum when the maximum is excluded. Node 3 thus remains the maximum after the 
first iteration. By induction, it will remain the maximum over all iterations and it re- 
sponded as expected. [Ref. 4] 

The remainder of the proof of convergence depends on demonstrating that the out- 
put of node 3 is never driven to zero, but the outputs of all other nodes are. When 
Equation 3-12 is satisfied, inhibit) is always less than the average value of all other node 
outputs. The inhibition to node 3 will thus be less than the average of the output of all 
nodes. Whenever a maximum exists, this inhibition will always be less than the current 




(3-17) 





-c, i # j, c < — , 0 < ij < M - 1 



(3-18) 
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output of node 3 because the maximum of a set of positive numbers is always greater 
than the average. The output of node 3 will not be driven to zero while any other nodes 
have nonzero outputs. After all other node outputs are driven to zero, the inhibition to 
node 3 drops to zero, and the output of node 3 remains constant. The output of all other 
nodes will always be driven to zero because the inhibition to these nodes remains posi- 
tive on all iterations and approaches a positive constant as time increases. In practice, 
the maxnet will still converge and find the maximum when each weight w tJ is set to 
\ P^ us a sma H random component. This forces the net to find a maximum when 
the inputs to all nodes are identical. As a matter of fact, this discussion can be general- 
ized to all input patterns. [Ref. 4] 

Then, we presented another input pattern which is of the digit "9". This pattern was 
noise corrupted in similar fashion as for digit ''3". The purpose of this simulation was to 
see if the net will behave as discussed earlier and if the number of iterations necessary 
for a successful convergence is function of the input pattern and how much noise dis- 
turbed it is. So. for the noisy pattern of digit "9", the response of the net is as provided 
in Figure 26. 

For more clarity, we are going to illustrate the response of the net in the following 
graphs for successive iterations. Each plot corresponds to the response of the net at the 
corresponding number of iterations, where the class output node values are decreasing 
from an iteration to the next. By inhibition, all of them will be driven to zero, some faster 
than others, except for the class output node corresponding to that to the stored 
exemplar that best matches the unknown input pattern. Then, this output node value 
will remain constant throughout future iterations while the zero valued output nodes 
will remain at zero. 
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Figure 26. Response of the Hamming net to the second input pattern 
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Figure 27. The output of the Hamming net at t = 1 for digit "9" 
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Figure 28. The output of the Hamming net at t = 2 for digit " 9 " 
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Figure 29. The output of the Hamming net at t = 3 for digit " 9 " 
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Figure 30. The output of the Hamming net at t = 4 for digit "9" 



43 




Figure 31. The output of the Hamming net at t = 5 for digit "9" 
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Figure 32. The output of the Hamming net at t = 6 for digit "9' 
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These graphs have proven once more, the net's mechanism of convergence described 
earlier. After 7 iterations, the net has effectively converged to the correct pattern of digit 
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"9", which output node was the only nonzero node while all the others were driven to 
zero by the convergence process of the Hamming net. Then, seven iterations were used 
to converge to digit "9" in comparison to eight to recognize digit "3". These simulations 
were for noise-corrupted input patterns, now we resimulated the net but this time using 
the perfect pattern of digit "2". The response of the net is provided in Figure 34. 
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Figure 34. Response of the Hamming net to the perfect input pattern 
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Even for the perfect input pattern of digit "2", the net took 7 iterations to success- 
fully converge to the correct response. So, the number of iterations is definitlv not 
function of the input pattern and how noise disturbed it is but with how many stored 
patterns it shares many similarities (almost same distribution of + 1 and —1). The num- 
ber of iterations necessary for a successful convergence depends on the output node 
values at the first iteration. The bigger magnitude these values have, the more similar 
their respective patterns are to the input and the more iterations will be necessary to 
drive them to zero by inhibition except for the correct output node. 

In our last simulation with digit "2" as a perfect input pattern, after the fourth iter- 
ation, only those output nodes corresponding to class "2" and class "6" were the only 
nonzero valued nodes. We can conclude, from this, that the input pattern, after the 
fourth iteration, is very close to the patterns of digit "6" and "2", but more closer to "2" 
than "6" because of its higher output node value. The network took two more iterations 
to drive the output node of the "6" to zero. Then, the inhibition to node " 2 " drops to zero 
and the value of its output node remains constant. The input pattern is said to be digit 
"2" pattern. 

This discussion can be generalized to explain the previous behavior of the network 
to digit "3" and digit "9" as input patterns. In all cases, the output node with the higher 
magnitude, after the first iteration is completed, is always the correct node but the con- 
vergence mechanism of the net is not to make decisions at this stage. 
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IV. THE CARPENTER / GROSSBERG NET 



A. GENERALITIES : 

Classified as a self-organizing neural net, the Carpenter / Grossberg net self- 
organizes and self-stabilizes its recognition codes in response to arbitrary sequences of 
binary input patterns. Top-down attentional and matching mechanisms are critical in 
self-stabilizing the code learning process. The architecture embodies a parallel search 
scheme which updates itself adaptively as the learning process proceeds. After the 
learning process has self-stabilized, the search process is automatically disengaged. 
Thereafter, input patterns directly access their recognition codes without any search. 
Thus, recognition time does not grow as a function of code complexity. 

A novel input pattern can directly access a category if it shares invariant properties 
with the set of familiar exemplars of that category. These invariant properties emerge in 
the form of learned critical feature patterns, or prototypes. The architecture possesses a 
context-sensitive self-scaling property which enables its emergent critical feature patterns 
to form. They detect and remember statistically predictive configurations of featural el- 
ements which are derived from the set of all input patterns that are experienced. Four 
types of attentional processes (priming, gain control, vigilance, and intermodal compe- 
tition) are mechanistically characterized. Top-down priming and gain control are needed 
for code matching and self-stabilization. Attentional vigilance determines how good the 
learned categories will be. If vigilance increases due to an environmental disconfirma- 
tion, then the system automatically searches for and learns the best recognition catego- 
ries. [Ref. 6] 

This chapter develops a theory of how recognition codes are self-organized by a 
class of neural networks whose qualitative features have been used to analyse data about 
speech perception, word recognition and recall, visual perception, olfactory coding, 
evoked potentials, thalamocortical interactions, attentional modulation of critical ter- 
mination, and amnesia. These networks comprise the adaptive resonance theory (ART) 
characterized as a system of ordinary differential equations. 

B. IMPLEMENTATION OF THE CARPENTER / GROSSBERG NET : 

The neural network that will be discussed in this chapter is known as an ART sys- 
tem, after the adaptive resonance theory introduced by Grossberg [Ref. 7], see Appendix 

C. Recently, ART networks have been further studied and their dynamic properties 
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have been derived in a series of theorems. These theorems predict both the order of 
search, as a function of the learning history of the net, and the asymptotic category 
structure self-organized by an arbitrary 7 binary input sequence. 

The operation of the ART system discussed in Appendix C will be used to develop 
a neural net known as the Carpenter / Grossberg net, using neural net components, 
which will form clusters and is trained without supervision. The net can learn from input 
patterns and later differentiate between new and stored (learned) patterns. If the new and 
unknown input pattern is classified as a previously learned pattern at a certain level of 
vigilance, it will be ignored, but if it is not, it will be added as a new pattern by the net. 
This process is repeated for all input patterns. The number of learned patterns thus 
grows with time and depends strongly on the level of vigilance (threshold) used to com- 
pare input patterns to the already stored ones. 

The operation of the Carpenter / Grossberg net which forms clusters (learned pat- 
terns) and is trained without supervision is given in eight steps : [Ref. 2] 

Step 1. Initialization 



Set p , 0 < p < 1. 

In these equations b„(t) is the bottom-up and t 0 {t) is the top-down connection weight 
between input node i and output node j at time t as shown in Figure 35. These weights 
define the exemplar specified by output node j. The fraction p is the vigilance threshold 
which indicates how close an input must be to a stored exemplar to match. 

Step 2. Apply New Input 

Step 3. Compute Matching Scores 




(4-1) 




(4-2) 



0 < / < iY - 1 , 0 <>j ^ M - 1 




(4-3) 
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In this equation fij is the output of node j and x, is element i of the input pattern 
which can be 0 or 1. 



OUTPUT 




Figure 35. The major components of the Carpenter / Grossberg classification net 
[Ref. 2] 



Step 4. Select Best Matching Exemplar 

,u ■ = max {nj} (4 - 4) 

j 

This is performed using extensive lateral inhibition as in the maxnet. 

Step 5. Vigilance Test 

,v-i 

Ml = (4-5) 

0 

iV-1 

117-^- All = Yt l/Xi (4-6) 

i= o 
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(4-7) 



If YES then GO TO Step 7, otherwise GO TO Step 6 
Step 6. Disable Best Matching Exemplar 

The output of the best matching node selected in Step 4 is temporarily set to zero 
and no longer takes part in the maximization of Step 4. Then go to Step 3. 

Step 7. Adapt Best Matching Exemplar 



Step 8. Repeat by Going to Step 2 

(First enable any nodes disabled in Step 6) 

After initialization of the net and presentation of an unknown input pattern, 
matching scores are computed using feed-forward connections. The node corresponding 
to the exemplar with the highest matching score is selected using lateral inhibition 
among the output nodes as in the maxnet (Hamming net), where each output node 
corresponds to a stored exemplar. This net differs from the Hamming net in that feed- 
back connections are provided from the output nodes to the input nodes and elements 
of both inputs and stored exemplars take on only the values 0 and 1. 

The selected exemplar, from the highest matching score, is then compared to the 
input by computing the ratio of the dot product of the input and the best matching 
exemplar (number of 1 bits in common) divided by the number of 1 bits in the input. If 
the ratio is greater than a threshold value (vigilance) which was set at the initialization 
of the net (Step 1 of the algorithm), then the input is considered to be similar to the best 
matching exemplar and that exemplar is updated by performing a logical AND opera- 
tion between its bits and those in the input. On the other hand, if the ratio is less than 
the vigilance threshold, the output node with the highest matching score is temporarily 



y('+ 0 = y(0 -*7 



(4-8) 



fy(M-l) = 




(4-9) 



52 



set to zero, disabled by provided mechanisms. The same input pattern is presented again 
to the net for another test. The cycle continues until one stored exemplar matches the 
input or this pattern is considered to be different from all stored exemplars and it is 
added as a new one. Generally, when the first test fails the input is a new exemplar. Each 
additional exemplar requires one output node and 2N connections to compute matching 
scores. 

The vigilance threshold, which ranges between 0.0 and 1.0, determines how close a 
new pattern must be to a stored exemplar in order to be similar. A value near 1.0 means 
a close match is necessary and smaller values accept a poorer match. 

C. SIMULATION OF THE CARPENTER / GROSSBERG NET : 

Using the Fortran program provided in Appendix D, we simulated the behavior of 
this net. For this simulation, a vigilance threshold of 0.9 was choosen, which means that 
an input pattern must be very close to a stored exemplar to be considered similar. The 
patterns used in this simulation were of the letters "C", "E", "F" and were choosen to be 
of 64 element representation (matrices of 8 by 8). In all the figures provided in this 
discussion, we have made a black pixel to correspond to an element of value 1 and the 
white pixel for the value of 0. The actual input patterns used are provided in Appendix 

D. 

Initially, the storage memory of the net was empty. To train this net, an input pat- 
tern representing the letter "C" was presented first and it was automatically stored as the 
net starts to learn. Now internal connection weights of the net are altered to form an 
internal exemplar that is identical to the letter "C" and we have the first output node of 
the net. In the same fashion, every learned pattern will be stored as 2N connection 
weights and one output node is added to the net. These 2\ connections weights will 
form an internal exemplar for the respective pattern stored. 
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In the storage memory of the net, we have only one stored exemplar : 



After, an input pattern representation of the letter "E" was applied. The response 
of the net to this input was : 



BECAUSE, THE RATIO IS LESS THAN THE VIGILANCE THRESHOLD 
THE INPUT PATTERN IS CONSIDERED TO BE DIFFERENT FROM 
ANY EXEMPLAR PATTERN STORED. THIS INPUT PATTERN IS 
THEN STORED WITH THE OTHERS AS A NEW EXEMPLAR PATTERN. 



Here "E" was compared to "C" as described in Step 5 of the clustering algorithm of 
the net and since the ratio was less than the vigilance threshold we now have two stored 
exemplars. 




For a new input pattern representation of the letter "F", the response of the net was 
the same as for "E". Here the input pattern was compared to both stored exemplars, but 
at all times the ratio was less than the vigilance threshold. The input pattern of "F" is 
then added as a new exemplar leading to three stored patterns. 
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At this point, we will try something different. We will present a noisy version of the 
letter "F" with a missing black pixel in the upper edge as shown in Appendix D. The 
reaction of the net was : 



BECAUSE, THE RATIO IS GREATER THAN THE VIGILANCE 
THRESHOLD, THE INPUT PATTERN IS CONSIDERED TO 
MATCH A STORED PATTERN WHICH IS UPDATED BY 
PERFORMING A LOGICAL 'AND' OPERATION BETWEEN 
ITS BITS AND THOSE OF THE INPUT PATTERN, AND 
THE NEW UPDATED PATTERN WILL LOOK LIKE: 



In this part of the simulation, the input pattern is found to match the stored 
exemplar of "F" because the ratio of the vigilance test was found greater than the vigi- 
lance value, i.e., the two patterns have many elements in common. The result was a 
degraded "F" due to the AND operation performed during the updating. Now in the 
memory of the net we still have three patterns with some changes in the pattern of the 
letter "F" : 




Presenting an even more noisier version of the pattern of "F" given in Appendix D, 
the reaction of the net was : 



BECAUSE, THE RATIO IS LESS THAN THE VIGILANCE THRESHOLD 
THE INPUT PATTERN IS CONSIDERED TO BE DIFFERENT FROM 
ANY EXEMPLAR PATTERN STORED. THIS INPUT PATTERN IS 
THEN STORED WITH THE OTHERS AS A NEW EXEMPLAR. 

Here the input pattern was compared first to the stored pattern of the noisy "F", but 
the ratio was less than the vigilance value. Then it was compared to the other stored 
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exemplars, one in each cycle, but the ratio was still less than the vigilance value. The 
input pattern is then considered different from existing exemplars and it is added as a 
new one in the memory of the net. At this point, we have four stored patterns : 



■■ ■■■■ ■■■■ 



These results illustrate the inaccuracies of this net in a noisy environement. For a 
vigilance value of 0.9, we took a stored pattern and changed some of its elements simu- 
lating the presence of small amount of noise in the channel. Then, presenting it again 
as an input pattern has made the net take it as a new pattern to be stored with the cor- 
rect version. Besides the noise, the value of the vigilance test can also alter the behavior 
of the net as we are going to show. Using the same input pattern sequence as before, 
we are going to simulate the clustering algorithm of the net, but this time with 0.7 as the 
vigilance threshold. Starting by presenting the pattern of the letter "C" as the input, the 
net automatically stored it in its empty memory as the net starts to learn. Now internal 
connections weights of the net are altered to form an internal exemplar that is identical 
to the letter "C". 

In the storage memory of the net, we have only one stored exemplar : 



Then, an input pattern representing the letter "E" was presented to net. The response 
of the net was: 

BECAUSE, THE RATIO IS GREATER THAN THE VIGILANCE 
THRESHOLD, THE INPUT PATTERN IS CONSIDERED 
TO MATCH A STORED PATTERN WHICH IS UPDATED BY 
PERFORMING A LOGICAL 'AND' OPERATION BETWEEN 
ITS BITS AND THOSE OF THE INPUT PATTERN, AND 
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THE NEW UPDATED PATTERN WILL LOOK LIKE: 



■ ■ ■ ■ 



Here, comparing the input pattern "E" to the stored "C" as described in Step 5 of the 
clustering algorithm. The ratio was found greater than the vigilance value (0.7). The re- 
sult was a degraded "C", as shown, due to the AND operation performed on its bits 
during the updating. In the memory of the net, the degraded pattern of "C" is stored 
instead of the initial pattern. Now we still have only one stored pattern. 

For the pattern of the letter "F" as an input, the net responded with the following 
message: 



BECAUSE, THE RATIO WAS LESS THAN THE VIGILANCE THRESHOLD 
THE INPUT PATTERN IS CONSIDERED TO BE DIFFERENT FROM 
ANY EXEMPLAR PATTERN STORED. THIS INPUT PATTERN IS 
THEN STORED WITH THE OTHERS AS A NEW EXEMPLAR. 



After comparing the input pattern of "F" to the stored and degraded pattern of "C" 
as in the vigilance test, the ratio was found to be less than the vigilance threshold which 
results in another stored pattern : 



■ ■ ■ 



Now, we are going to present the noise corrupted pattern of "F". The net's response 

was: 



BECAUSE, THE RATIO IS GREATER THAN THE VIGILANCE 
THRESHOLD, THE INPUT PATTERN IS CONSIDERED 
TO MATCH A STORED PATTERN WHICH IS UPDATED BY 
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PERFORMING A LOGICAL 'AND' OPERATION BETWEEN 
ITS BITS AND THOSE OF THE INPUT PATTERN, AND 
THE NEW UPDATED PATTERN WILL LOOK LIKE: 



The matching pattern was of the letter "F". At this point, checking the net's memory 
will reveal the storage of the two patterns : 



Presenting a more corrupted pattern of "F", the response of the net was as before. 
The input pattern was found similar to the first corrupted version of "F" (the ratio was 
greater than the vigilance threshold). The stored pattern of the corrupted "F" was again 
more disturbed after the AND operation was performed between its bits and the input 
pattern. The result, once again, replaced the previous version of the corrupted "F" in the 
memory of the net : 



■ ■ ■ ■ ■ 



The results of the two simulations show clearly how the noise and the vigilance 
threshold can affect the performance of the Carpenter / Grossberg net. We have seen the 
net performing well for perfect input patterns and when adding a small amount of noise 
it behaves totally different. With no noise, a lower vigilance value can make the net 
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consider two different patterns to be similar. We have seen this in the second simulation 
when we have presented the pattern "E", which was mistakenly considered similar to "C" 
for 0.7 vigilance. On the other hand, a higher vigilance threshold can make the net con- 
sider two patterns, which are most similar, to be different. Thus, this net should not be 
used in a noisy channel with a high vigilance value; otherwise, the number of stored 
patterns will grow rapidly in time as input patterns are continuously presented until all 
available nodes are used up. A proportional adaptation of the vigilance threshold to the 
existing noise in the channel can make the net to perform perfectly during training and 
testing. 



59 



V. NEURAL NETWORK AS A BINARY MAXIMUM-LIKELIHOOD 

SEQUENCE ESTIMATOR 



A. GENERALITIES : 

Bandwidth-efficient data transmission over telephone and radio channels is signif- 
icantly improved by the use of adaptive equalization to compensate for the time 
dispersion introduced by the channel. 

During the last two decades, a steady research effort has produced a rich body of 
theory in the field of adaptive equalization and the more general field of adaptive re- 
ceivers. From this work, a class of nonlinear receivers referred to as maximum-likelihood 
sequence estimation receivers have emerged as front-runners with respect to error rate 
performance. However, the high degree of computational complexity of the optimal 
maximum-likelihood receivers has prohibited their use in many applications. It will be 
shown that neural networks can be used to implement the maximum-likelihood sequence 
estimation and that the networks offer an attractive alternative for implementation. 
[Ref. 3] 

Intersymbol interference caused by the bandlimiting effect of the channel is re- 
viewed. A maximum-likelihood receiver designed to detect data symbols in the presence 
of intersymbol interference and additive Gaussian noise is considered and the theory 
behind maximum-likelihood sequence estimation is reviewed. 

The maximum likelihood sequence estimation function is mapped onto a neural 
network structure. A neural network based receiver structure will be described which can 
be used for stationary or time- varying channels. The MLSE neural network will be sim- 
ulated on the Mainframe and some results of its simulation will be presented. 

B. MAXIMUM-LIKELIHOOD SEQUENCE ESTIMATION : 

Consider a baseband synchronous data communication link used to transmit a se- 
quence of numbers called data or information symbols, denoted by 

{... , , Q[ , j , •••} 

The symbols are independent and can, with equal probability, be either +1 or —1. 
Let M be the number of data symbols in a transmitted sequence and assume 
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transmission starts at time t = 0 and ends at time t = MT. The receiver will observe the 
signal y(t) during the time interval starting at t = 0 and ending at t = t f , where 

t f > {M + L)T 

where L is the channel memory in units of T. 

Denote the time inteval 0 -+ t f by By its definition, a maximum-likelihood re- 
ceiver determines {aj as the best estimate sequence { a n } that maximizes the likelihood 
function p\_y{t ) , t e /, | {a„}] given by 

p[y{t), te I r | {a„}] ~ exp j n{t x | {a„}) K^\t x - t 2 ) n(t 2 | {a„}) dt x <* 2 J(5 - 1) 

where K~'(t ) is the inverse of the noise autocovariance function K n (t) and 



n{t | {«„}) = y(t) ~Yj a k w - KT > (5 - 2) 

k= I 

where h(t) is the impulse response of the matched filter used in the adaptive maximum- 
likelihood receivers (Figure 36). Rearranging Equation 5-2, 

M 

y(') = Yu a k h ~ KT > + I {««}) ( 5 - 3 ) 

k= 1 



A block diagram of an adaptive maximum-likelihood receiver for the data trans- 
mission model described by Equation 5-3 is illustrated in Figure 36. 




Figure 36. Adaptive Maximum-Likelihood Receiver [Ref. 3] 
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The impulse response of the matched filter in Figure 36, designed to improve the 
signal-to-noise ratio, is given by 

g{t) = (5-4) 

where * denotes the convolution function. Substituting Equation 5-2 into Equation 5-1, 
expanding the terms in the braces and considering only terms that depend on {«„}, yields 

{ M M M -) 

^2 a t z, - YZ* s ‘~ k a *f ( 5 - 5 ) 

(=1 l=\k= l 3 



where, 






= j j h ^ X ~ ^ K 1 dt \ dt 2 

= J J h(t l - iT ) FT 1 (fj - t 2 ) h{t 2 - KT ) dt x di 2 



= s_i l = k — i 



and 



m = k -/) 



(5-6) 



(5-7) 



(5-8) 



The quantities z„ and s, can be interpreted as sample values taken at the output of 
the matched filter, where z n is obtained by sampling the output z(t) of the matched filter 
once every T seconds and s, 's account for the combined response of the transmission 
channel and matched filter. The s/s are symmetric and s, = 0 for |/| > L. [Ref. 8] 
Under maximum-likelihood criteria, the estimated sequence is that for which ex- 
pression 5-5 is maximized. Since 5-5 is monotonically increasing function of the term in 
braces, given by 



St M M 

J \t({ a n)) = Y a ‘ Zi ~ YjY. ] a ‘ S i-k a k- 



/=! 



Mk= 1 



(5-9) 
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maximizing Equation 5-5 is equivalent to maximizing Equation 5-9. The notation 
indicates the cost function for the sequence a x , a 2 , ... , a M . Equation 5-9 will be 
referred to as the MLSE cost function. [Ref. 3] 

The estimation procedure using direct evaluation of the MLSE cost function re- 
quires that Equation 5-9 be evaluated for all the possible sequences of length M that can 
be formed from data symbols + 1 and —1. Thus Equation 5-9 must be evaluated 2 M 
times to obtain an estimate of the sequence {«„}. To perform the estimate in real time, 
which is required by most communication links, the 2 M computations of Equation 5-9 
must be performed in MT seconds. In most cases, direct evaluation of tne MLSE cost 
function is too computation intensive to be of practical use. [Ref. 3] 

The number of computations required can be greatly reduced by the use of the 
Viterbi algorithm [Refs. 8,9], which requires on the order of 2 L ~ 1 multiply-and-add oper- 
ations during each signaling inteval T. 

C. NEURAL NETWORK : 

Any neural network has, as discussed before, parallel input channels, parallel output 
channels and a large amount of interconnections between the neural processing ele- 
ments. Figure 37 illustrates the general structure of a Hopfield neural network. The 
processing elements (nodes), or neurons, are modelled as amplifiers in conjunction with 
feedback circuits comprised of wires, resistors and capacitors organized so as to model 
the most basic computational features of neurons, namely axons, dendritic arborization 
and synapses connecting the different neurons. [Ref. 10] 

The model considered here for implementation of the MLSE neural network is that 
due to Hopfield and Tank [Ref. 3]. 

The amplifiers have sigmoid monotonic input-output relations, as shown in Figure 
37. The function v(r) = g [w(r)] which characterizes this nonlinear input-output relation 
describes the output voltage v(t) due to an input voltage u(t). The time constants of the 
amplifiers are assumed negligible. However, like the input impedance caused by the cell 
membrane in a biological neuron, each amplifier j has an input resistor p, leading to a 
reference ground and an input capacitor c r These components partially define the time 
constants of the neurons and provide for integrative analog summation of the synaptic 
input current from other neurons in the network. 

In order to provide for both excitatory and inhibitory synaptic connections between 
neurons while using conventional electrical components, each amplifier is given two 
outputs, a normal ( + ) output and an inverted (— ) output. 
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f 









u(«) 



Figure 37. Hopfield Neural Network (Ref. 3] 



The minimum and maximum outputs of the normal amplifier are taken as 0 and 1, 
while the inverted output has corresponding values of 0 and —1. A synapse between 
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neurons is defined by a conductance w t . which connects one of the two outputs of am- 
plifier j to the input of amplifier i. This connection is made by a resistor of value 
' , . If the synapse is excitatory {w tj > 0), this resistor is connected to the 



U-'.. 



normal (+ ) output of amplifier j. For an inhibitory synapse (iv, ; < 0). it is connected to 
the inverted (— ) output of amplifier j. The matrix w tJ defines the connectivity among the 
neurons. The net input current to any neuron i (and hence the input voltage u,) is the 
sum of the currents flowing through the set of resistors connecting its input to the out- 
puts of the other neurons. [Ref. 9] 

The set of differential equations describing the dynamics of the neural network 
shown in Figure 37 with M neurons are given by 



didt) v~ ’ uh) 

c j ~ nr = L vv * v *w - ~t~ + 7 * f - - v/ ( 5 ~ 10 ) 

k = 1 1 

where v,(r) = g , [w,(r)] and R, is the parallel combination of p, and the R tJ 's, and C t is the 
capacitance of amplifier j. 



M 




j= i 



(5-11) 



For simplicity, we assume that g,[.] = g[.]. R, = R and C, = C, independent of i. 

w ;A / ? 

Dividing Equation 5-10 by C and redefining w ik = and /, = — . the equations of 
motion become : 



du ft) 
dt 



M 

^j W ik v fc(0 



*=1 



“X0 



+ h 



i = 1 , 






(5-12) 



where t = RC is the time constant of the circuit. [Ref. 9] 

In the Hopfield net operation, it was shown that the equations of motion for a net- 
work with symmetric connections (w 0 — w Jt ) always lead to a convergence to stable 
states, in which the outputs of all neurons remain constant. Also, when the width of the 
amplifier gain curve in Figure 37 is narrow, the stable states of a neural network com- 
prised of M neurons are the local minima of the quantity 



65 



(5-13) 



E = - X/'W 7 ' ” ~2 W * V *W 




/=] 



When high amplifier gain is used, the minima occur only at the corners of an M- 
dimensional hypercube defined by v, = +1 or —1. 

D. MAPPING OF MLSE ONTO A NEURAL NETWORK : 

Maximizing the MLSE cost function described by Equation 5-9 is equivalent to 
minimizing the following expression 



where a ,' s (of only + 1 and —1) values which minimize Equation 5-14 are unknown. The 
r,'s and s,_f s are known. Comparing Equations 5-13 and 5-14 and equating variables 
as follows , 



reveals that the two expressions are identical under these substitutions. From Equation 
5-7, we recall that s, = Therefore, 



which satisfies the synaptic interconnection symmetry condition. The synaptic intercon- 
nections for the neural network are determined by the coefficients which describe the 
combined response of the channel and matched filter. Let W denote a matrix of synaptic 
connections \v, k . Then the synaptic connection matrix W for the network is given in 
Figure 38. 

The externally supplied input current for each neuron, /„ is determined by observa- 
tion z, , 1 < i < M. With the input voltage u,(t ) initially at zero, the input sequence is 
applied to the network. After the network settles the estimated sequence {a„} is read from 
the output of the neural amplifiers. A diagram of the MLSE neural network is shown in 
Figure 39. [Ref. 3] 




(5 - 14) 



2 z t I - t , 2 w ik , a L l i(0 



n ik - s i—k - s k—i w ki 



66 




hhi 2*m_i 2 zj 



• • • 



NEURAL NETWORK 
INTERCONNECTIONS: W ,V« 




Uhl OM-l 0| 



Figure 39. MLSE neural network. [Ref. 3] 



This development assumed that the transmission channel is stationary, which im- 
plies that the s/s describing the combined channel and matched Filter response do not 
change with time. Often, this is an unrealistic assumption. [Ref. 3] 

The MLSE cost function given by Equation 5-14 can be written for the time- varying 
channel as 
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(5 - 15) 



M M M 

= ~ Yj 1 a ‘ 2 ‘ + YY a ‘ S i-k a k 

i= 1 i=l /<=1 

From Appendix E and using Equation 5-15, the parameters for the MLSE neural 
network are given by [Ref. 3] 

2 z, = I,, -2 s’i_ k = ~(sV k + s£i) = w lk , a, = v-(r) (5-16) 

E. NEURAL NETWORK MAXIMUM-LIKELIHOOD RECEIVER : 

A block diagram of the adaptive maximum- likelihood receiver incorporating the 
neural network for MLSE is shown in Figure 40. Registers A, , R 2 , ... , R M form a shift 
register used to store the M observations. With all amplifier inputs w, , i = 1, ... , M, 
initially at zero, switches SIV, , / = 1, ... , A/, are simultaneously closed and the network 
is allowed to settle. The output of each neural amplifier is applied to the input of a de- 
cision device which outputs a +1 or —1 for a positive or negative input respectively. 
Once the network has settled, the estimated sequence is read at the output of the deci- 
sion devices as shown in Figure 39. [Ref. 3] 

In some cases, the length of the network, M, will be considerably less than the total 
number of data symbols in a transmitted sequence. For example, suppose the transmit- 
ted sequence consists of A' x M data symbols, one approach is to load the first set of M 
observations and estimate the corresponding data symbols. After the estimate is ob- 
tained. the second set of observations would be loaded and the second set of data sym- 
bols estimated. The procedure would be performed a total of K times to obtain an 
estimate of the entire transmitted sequence. The primary problem with this approach is 
that it does not take into account the truncation of the observation sequence. [Ref. 3] 
The effect of the truncation can be described by considering the role of the obser- 
vations in the estimation. Let z, (<> , 1 < k < M, denote an observation applied to the ex- 
ternal input of neural amplifier k, where is the i' k observation from the received 
sequence. 
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Figure 40. Neural Network Based Maximum-Likelihood Receiver [Ref. 3] 



Since the channel memory is L, all information concerning the identity of data 
symbol a, is contained in observations: [Ref. 3] 
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Ak-L) Jk-L+ 1 ) 

Z i—L ’ z i—L,-\ m 1 » ••• » 



Jk+L) 

-i+L 



for k in the interval 



L < k <M-L+ 1 (5-17) 

All the observations containing information about the data symbol estimated by 
neural amplifier k are available to the network. On the other hand, for k in the intervals 

1 < k < L & M - L + 1 < k < M (5-18) 

some of the observations containing information about the data symbol estimated by 
neural amplifier k are not available to the network. Therefore, one would expect more 
errors to occur in estimates a m for k in the interval given by Equation 5-18 than in the 
interval given by Equation 5-17. [Ref. 3] 

This problem can be solved by overlapping the sequences used for each estimation 
iteration. Assume a set of M observations have been received and the network has 
produced a set of M data symbol estimates. Rather than accept all M estimates as valid, 
only estimates from neurons L+ 1 through M-l are taken as valid. This, of course, cor- 
responds to the estimates based on complete information about the symbols being esti- 
mated. From this set of observations, the observations in shift registers M, M-l, ..., p+ 1 
are saved, where 2 L < p < M — 1. A new set of p observations are shifted into the shift 
registers and the network performs another estimation. Essentially, this procedure 
amounts to shifting in p rather than M new observations after each estimation cycle. 
[Ref. 3] 

F. SIMULATIONS AND RESULTS : 

The neural network based MLSE receiver structure was implemented and simulated 
on the Mainframe. The program used is a self-driving program, provided as Appendix 
F. The network was simulated by numerically solving the set of M differential equations 
of Equation 5-12. The differential equations solver used in the simulation was the sub- 
routine DGEAR of the IMSL library. 

Then the M output values of DGEAR subroutine were passed through their re- 
spective neural amplifiers. The input-output function of the neural amplifiers was im- 
plemented as a hyperbolic tangent function. 

!,■(/) = - tanh[G i/,(')] ‘ •••> M (5-19) 
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where G is the gain constant. Increasing G increases the slope of the input-output curve 
in the transition region and reduces the width of the region (see Figure 37) [Ref. 3]. 

The transmission channel impulse response is modeled by a finite response square 
cosine function given by [Ref. 10] 



ry x 2 7t t 

{ F(a) cos 

0 

0, 

where the multiplica tiveterm F(ot) is included to model the time- varying channel. For the 
stationary channel simulations, F(y.) is taken as 1. Channel interference also includes 
additive White Gaussian noise n(t). The combined response of the channel and matched 
filter is then [Ref. 10] 

+ -^ 2 -sin(2*|/|7' 0 )l |/| < T„ 

4 J (5-21) 



s(t, a) = 



f 2 (y) 

2 A’n 



1(7-0- Ul) 



i,l 2 nt 

l+ T C0 \— 



Ul > To 



- Ul < 



_2_ 

2 



(5 - 20) 



f > 



where N 0 denotes the single-sided spectral density of the additive White Gaussian noise 
n(t), and T 0 is the time duration of the intersymbol interference. Equations 5-20 and 5-21 
are sampled at intervals of T seconds, where T is the bit duration, to generate the L + 
1 discrete time channel coefficients (h,(a)'s) and 2L + 1 discrete time coefficients de- 
scribing the combined response of the channel and matched filter (s,(a)'s). VLSI imple- 
mentation using sequential processing techniques have been reported for data rates up 
to 2400 bits/second [Ref. 3]. The channel memory L is given by 



L = 




(5 - 22) 



j 

Actually, L is the largest integer less than or equal to —jr but in this simulation we 
T 

are going to take L as -jr. 



Using the coefficients generated by sampling Equations 5-20 and 5-21 and assuming 
a stationary channel ( F(a ) = 1) and baseband transmission model, the received samples 
(v','s) are generated by the expression 
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/ = 1 , .... M 



(5-23) 



A 1 

yi = Yj a « h ‘-k + 

k= 1 

where y t replaces y(t) at t = iT, h,_ k replaces h(t) at t = (i - k)T and n, is a sample of the 
additive White Gaussian noise n(t) at t = iT. For this simulation, the noise samples are 
generated by a Gaussian random number generator GGNML of the IMSL library. The 
data symbols (erfs) which are + 1 and —1, are generated with equal probability using a 
uniform random number generator GGUD of the IMSL library. Then, the observations 
(z„'s) are generated by the expression 



M 



z n = Jj' k h k~n 



n = 1, 



M 



(5 - 24) 



k = 1 



The s*'s and z ’ s generated by the Equations 5-23 and 5-24 are substituted into 
Equation 5-16 to define the parameters of the neural network. A gain factor G of 10000 
was used for the simulations because of the very small output numbers of the differential 
equations solver DGEAR. Each simulation started with zero initial conditions and the 
computations were stopped after simulation of 5r seconds (5 time constants). The esti- 
mated sequence, the M outputs of the neural amplifiers (Figure 39), was then compared 
to the transmitted sequence and the total number of errors were recorded. Also, the es- 
timated sequence was compared to an estimate obtained by direct computation of the 
MLSE cost function and the number of data symbols which differed between the two 
estimated sequences was recorded. 

Table 1 lists the number of neurons (M) used in the simulation, the network time 
constant (r), the channel memory L which was taken to be 2 for all the simulations, the 
signal-to-noise ratio (SNR) given at the output of the matched filter and is computed 
by 



SNR 



111 

4 An 



the number of data (p) shifted into the registers at each simulation step, the number of 
symbols transmitted (X), the error data for each simulation. The last column of Table 
1 lists the number of data symbol estimates which differed between the neural network 
estimates and direct computation of the MLSE cost function. 
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No. of 
neurons 
(M) 


Time 

con- 

stant 

(msec) 


Chan- 
nel 
mem- 
on ( L) 


SNR 

(dB) 


P 


No. of 
symbols 
transmitted 

(N) 


Total 

Errors 


Total 

dilTer- 

ences 


9 


29.7 


2 


8 


6 


2500 


0 


0 


9 


11. S 


2 


12 


6 


2500 


0 


0 


9 


4.7 


2 


16 


6 


2500 


0 


0 


9 


1.9 


2 


20 


6 


2500 


0 


0 


17 


14.9 


2 


8 


12 


1500 


0 


0 


17 


5.9 


2 


12 


12 


1500 


0 


0 


17 


2.3 


2 


16 


12 


1500 


0 


0 


17 


1.0 


2 


20 


12 


1500 


0 


0 


25 


9.9 


2 


8 


18 


1000 


0 


0 


25 


3.9 


2 


12 


18 


1000 


0 


0 


25 


1.6 


2 


16 


18 


1000 


0 


0 


25 


0.6 


2 


20 


18 


1000 


0 


0 



Table 1. SIMULATIONS RESULTS FOR MLSE NEURAL NETWORK (STA- 
TIONARY CHANNEL) 



To simulate the time- varying channel, the multiplicative F(a) will be changed at each 
sampling instant. The value of F(a) is constrained to be in the interval 

0.8 < F{ a) < 1.0 

The random number generator GGUD of the IMSL library was used to generate 
uniformly distributed samples, A,, as described in the computer program of Appendix 
F. The generated samples A, are distributed in the interval 

-0.1 < A; < 0.1 

So at a certain sampling instant, say a = iT, the value of F{o.) is computed by 

F(iT) = 0.9 + A,- 

The transmission channel impulse response samples hf s and the combined response 
of the channel and matched filter sf s along with z, are the parameters describing the 
time-varying channel at a certain time t = iT. With the exception of this modification. 
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the simulator for the time-varying channel is identical to that for the stationary channel. 
Results of this simulation for the time-varying channel are listed in Table 2. [Ref. 3] 



No. of 
neurons 
(M) 


Time 

con- 

stant 

(msec) 


Chan- 
nel 
mem- 
ory (L) 


SNR 

(dB) 


P 


No. of 
symbols 
transmitted 

(N) 


Total 

Errors 


Total 

differ- 

ences 


9 


29.7 


2 


8 


6 


2500 


0 


0 


9 


11.8 


0 

X. 


12 


6 


2500 


0 


0 


9 


4.7 


2 


16 


6 


2500 


0 


0 


9 


1.9 


2 


20 


6 


2500 


0 


0 


17 


14.9 


2 


8 


12 


1500 


0 


0 


17 


5.9 


2 


12 


12 


1500 


0 


0 


17 


2.3 


2 


16 


12 


1500 


0 


0 


17 


1.0 


0 


20 


12 


1500 


0 


0 



Table 2. SIMULATION RESULTS FOR MLSE NEURAL NETWORK 
(TIME- VARYING CHANNEL) 



The neural network presented in this study can be thought of as an alternative to 
the Viterbi algorithm [Ref. 11] for computation of the MLSE cost function. Unlike the 
Viterbi algorithm implementation, the neural network does not require a vast amount 
of memory for storage. From the simulation results for the two transmission channel 
conditions for a channel memory of 2, we can conclude that the neural network can be 
used to estimate a transmitted sequence of binary data symbols. Comparing the esti- 
mates of the MLSE neural network and those of the direct computation of the MLSE 
cost function, we can say that The MLSE neural network does indeed perform the de- 
sired estimation. The amount of data provided by the simulations is far too little to make 
any final conclusions concerning the performance of the MLSE neural network. How- 
ever, the results are promising and indicate that the neural network may be an attractive 
alternative for implementation of MLSE for binary communications systems. [Ref. 3] 
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VI. CONCLUSION 



A. SUMMARY OF RESULTS : 

In this study, we have First made an introduction to the field of artificial neural 
networks. Then, we described the use of some neural networksin pattern recognition and 
classification using binary pattern elements. A computer program from an algorithmic 
approach for each one of these networks was constructed and used to simulate the op- 
eration of the net for different cases of input pattern. 

The Hopfield network was the first net we worked on. A simulation program 
imlementing the operation of this net as a content addressable memory for random input 
patterns was made. As a supervised network, the Hopfield net is only iterating between 
an input pattern and the ones that the teacher has already stored in its memory, showing 
that this net is a non-learning one. This net was simulated by presenting noise-corrupted 
or perfect input patterns. The response of the Hopfield net to each one of these input 
patterns was provided to show the iterations taken by the net to recognize and classify 
even noise-corrupted input patterns. By recognition and classification, we mean the net 
converges to one of the M stored patterns that best matches the input pattern, as long 
as the original pattern was stored in the net's memory prior to its use, otherwise a "no 
match" will occur. 

However, the number of stored patterns (M) is a limitation to the proper operation 
of the net as a classifier because of the convergence condition demonstrated by Hopfield, 
which states that the net will converge with high probability if M < 0.15/Y, where X is 
the number of elements or bits in each pattern. These bits are taking on + 1 and — 1 
values, for the + 1 and —1 states, respectively. 

The Hamming network is a classifier that calculates the Hamming distance to the 
exemplar of each stored class and select that class with the minimum Hamming distance 
to the specified input pattern. The Hamming distance is the number of bits in the input 
which do not match the corresponding exemplar bits. As a supervised network, we have 
first stored 10 exemplar patterns (M= 10) in its memory prior to its simulation. Simu- 
lating the operation of this network on the computer, we have seen that it effectively 
converges to the correct class for each input pattern. Even presenting noise corrupted 
input patterns, the net correctly converges to the correct class, as long as the original 
pattern was stored in its memory prior to the simulation, otherwise a "no match" will 
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occur. By convergence of the net. we mean the output nodes of the upper subnet (see 
Figure S) stop changing in time and only the output node corresponding to that 
exemplar class which best matches the input pattern, is a positive nonzero value. While, 
all the other output nodes were driven to zero by inhibition. In practice, the net will 
converge and find the correct class when each weight w,., connection weight from input 
i to node j in the lower subnet (see Figure 8), is set to y plus a small random 
component. Like the Hopfield net, the elements of the patterns used in these simulations 
were taking on + 1 and —1 values for the + 1 and —1 states, respectively. 

As a self-organizing (a non-supervised) neural net, the Carpenter / Grossberg net 
self-organizes and self-stabilizes its recognition codes in response to arbitrary sequences 
of binary input patterns. In its learning process, the net uses a threshold level called the 
vigilance value which determines how good the learned categories will be. If vigilance 
value increases due to an environmental disconfirmation, then the net automatically 
searches for and learns the best recognition categories. The Carpenter / Grossberg net 
is well known as an ART system, described in Appendix C, which forms clusters and is 
trained without supervision. This net can learn from input patterns and later differentiate 
between new and learned patterns. If the new and unknown pattern is classified as 
previousely learned pattern at a certain level of vigilance, it will be ignored, but if it is 
not, it will be added r.s a new learned pattern. This process is repeated as long as the net 
is learning. The number of learned patterns thus grows with time and depends strongly 
on the level of vigilance used to compare input to the already stored ones. The results 
of simulating this network showed clearly the importance of the vigilance threshold. The 
first simulation was done with a vigilance value of 0.9, which means that an input pat- 
tern must be very close to a stored exemplar to be considered similar. The result was 4 
patterns learned out of 6 input patterns because the net has taken two input patterns 
as an already learned one. Next, we have done the same simulation but this time with a 
vigilance value of 0.7. The results were 2 patterns learned out of 6 input patterns pre- 
sented. So higher vigilance threshold can make the net to consider two patterns which 
are most similar, to be different and lower threshold can make the net to consider two 
different patterns as similar. Thus the vigilance threshold, used in the learning process 
of this network, is the dominant factor in the operation of this net, which behavior de- 
pends strongly on it. A proportional adaptation of the vigilance level to the existing 
noise in the channel can make the net to perform perfectly during training and testing. 
The elements of the patterns used to simulate this net take on, contrary to the Hopfield 
and Hamming nets, the + 1 and 0 values for the + 1 and —1 states, respectively. 
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On the other hand, bandwidth- efficient data transmission over telephone and radio 
channels is significantly improved by the use of adaptive equalization to compensate for 
the time dispersion introduced by the channel. From the work done on adaptive re- 
ceivers, a class of nonlinear receivers referred to as maximum-likelihood sequence esti- 
mation receivers have emerged as front-runners with respect to error rate performance. 
However, the high degree of computational complexity of the optimal maximum- 
likelihood receivers has prohibited their use in many applications. It was shown that 
neural networks can be used to implement the MLSE and that these networks offer an 
attractive alternative for implementation. After mapping the MLSE onto a neural net- 
work, we have done some simulations on this network for stationary and time-varying 
channels. The results, even though they are not based on enough data to draw definitive 
conclusions, showed that the neural network may be an attractive alternative for imple- 
mentation of the MLSE for binary communications. 

B. NEURAL NETWORK TASKS : 

The field of neural networks include many different models designed to address a 
wide range of problems in the primary application areas of speech, vision and robotics. 
Most researchers focus on neural networks that perform those seven major tasks illus- 
trated graphically in Figure 41. These tasks include : [Ref. 1] 
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• Pattern classification: Classifiers are trained with supervision using labeled training 
data to partition input patterns into a pre-specified number of groups or classes. 
These could represent different objects for a visual image classifier. Inputs to a 
classifier may be binary as we have seen for the Hopfield and Hamming nets or 
continuous- valued. 

• Self-organization or Clustering: Self-organizing networks, like the Carpenter / 
Grossberg net, partition input examples into groups or clusters using unlabeled 
training data. This type of clustering or vector quantization is an efficient technique 
for reducing information that must be processed at higher levels with little loss in 
performance. It also makes good use of the large amount of unlabeled training data 
that is typically available in speech and vision problems. 

• Associative memory (storage and access): An associative, or content-addressable 
memory provides a complete memory item from a key consisting of a partial or 
corrupted version of the memory. For example, it might return a complete article 
citation from only the author's name or a complete image of a face from only the 
bottom half. 

• Sensor} - Data Processing (vision and speech): An enormous amount of realtime 
preprocessing is performed in the peripheral sensory vision and hearing centers. 
Neural networks can perform this function in real time using massive parallelism. 

• Computational Problems: Custom neural network architectures can be designed to 
solve specific computation problems, such as the traveling salesman problem and 
other constrained optimization problems, using nonlinear analog computation. 

• Nonlinear Mapping: Many neural networks can map a vector of analog inputs into 
an output vector using a nonlinear mapping function which can be learned from 
training data. These types of mappings are useful in many areas, including robot 
control and nonlinear signal processing. 

• Multi-sensor Automata: A number of complex, multi-module neural network 
automata have been built with visual input and a robot arm to manipulate objects 
in an environment. These automata demonstrate how an eye or camera can learn 
to scan a scene using self-supervision, how control of a multi-jointed arm and hand 
can then be learned using self-supervision, and then how the eye and hand can be 
coordinated to perform simple tasks. These automata also demonstrate how inputs 
from multiple sensors can be fused to provide classification performance better 
than could be achieved with a single sensor. 

C. CONCLUSIONS : 

From the study done by DARPA [Ref. 1], we can conclude that neural networks 
offer important new computational structures. Their real strength is derived from their 
ability to self-adapt and learn. If neural networks realize their full potential, they can be 
used for machine vision, speech recognition, signal processing, robotics and other ap- 
plications. 

Neural network research has matured greatly since the perceptron of 1950s, thanks 
to the development of advanced mathematical theories and new computer tools, and also 
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to a better understanding of a? eurobiolog}'. The hardware capabilities are limiting the de- 
velopment of important neural network applications. It is clear that if researchers are 
not provided with improved simulation and implementation capabilities, the field of 
neural networks will once again drift off into the wilderness. 
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oooooooooooooooooooooonoooooooooo 



APPENDIX A. PROGRAMING THE HOPFIELD NET WHEN USED AS 

A CLASSIFIER : 

Using Fortran as programing language, the previously described operation algo- 
rithm of the Hopfield net when used as a classifier was implemented with the mainframe, 
and used to run some simulations as described in the simulation paragraph of the 
Hopfield net. 
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THESIS RESEARCH 

HOPFIELD NET SIMULATION PROGRAM 
BY M. H. KHAIDAR 
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* 
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* 



* 

?v 
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* 









THIS PROGRAM WAS MADE TO IMPLEMENT THE HOPFIELD NETWORK * 

OPERATION ALGORITHM WHEN THIS NETWORK IS USED AS A * 

CLASSIFIER. AFTER THE INPUT PATTERN IS PROCESSED AS * 

DISSCUSSED BEFORE AND AFTER CONVERGENCE, THE OUTPUT WILL BE * 
COMPARED TO THE M (M = 8 IN THIS IMPLEMENTATION) EXEMPLARS * 
TO DETERMINE IF IT MATCHES AN EXEMPLAR EXACTLY. IF IT DOES, * 
THE OUTPUT IS THAT CLASS WHOSE EXEMPLAR MATCHED THE OUTPUT * 
PATTERN. IF IT DOES NOT THEN A "NO MATCH" RESULT OCCURS. 
DECLARATION OF VARIABLES: * 

PATT( I , S) = THE ITH ELEMENT OF THE STH STORED EXEMPLAR * 
T( I , J) = THE CONNECTION WEIGHT FROM NODE I TO NODE J* 
U(J,T) = THE OUTPUT OF NODE J AT TIME T * 

W(I) AND V(I) = THE ITH ELEMENT IN THE MATRIX COLUNM 

INPUT PATTERN W AND THE MATRIX COLUNM * 

OUTPUT PATTERN V 

MAT( 12 , 10) , V( 12 , 10) AND CMAT(12,10) = THE 12 BY 10 * 

MATRIX REPRESENTATION OF AN EXEMPLAR * 

CLASS(J) = THE MATRIX COLUNM OF THE JTH STORED PATTERN* 
N = THE NUMBER OF ELEMENT IN EACH EXEMPLAR 

M = THE NUMBER OF STORED EXEMPLARS * 

DIFF( 120 ,J) = THE DIFFERENCE BETWEEN THE OUTPUT MATRIX * 
COLUNM AFTER CONVERGENCE AND THE JTH STORED* 
PATTERN FOR CLASSIFICATION * 






INTEGER PATT( 120 , 8) , U(10,120), T( 120, 120), V(12,10), CLASS(120) 
INTEGER S, I, J, K, COUNT, MAT(12,10) 

REAL W( 120) , VEC( 120) , DIFF(120,8), CMAT(12,10), DMAT(12,10) 
CHARACTER* 1 TEMP ( 1 2 , 1 0 ) 

PRINT*,' HOPFIELD NETWORK IMPLEMENTATION' 

PRINT* , ’ ============================ === ' 

C 
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**************************************************************** 



* INITIALIZE WITH UNKNOWN INPUT PATTERN 

**************************************************************** 



N = 120 
M = 8 

0PEN(UNIT=1 , FILE='NINE1' , STATUS= ' OLD ' ) 

DO 5 1=1, N 

READ(1,*) W(I) 

U( 1 , I ) = W( I) 

5 CONTINUE 
PRINT*,' ' 

PRINT*,' ' 

PRINT 1 

1 FORMAT('THE UNKNOWN INPUT PATTERN TO THE HOPFIELD NETWORK, ' 

&' ON THE LEFT AS IM-'/ 

&' POSED ON THE NETWORK AND ON THE RIGHT IN A MUCH CLEAR ' 

&' REPRESENTATION'/ 

&' WHERE EVERY ( * ) REPLACES A 1 AND EVERY ( . ) REPLACES A -1: ') 
CALL VECMAT(W,MAT) 

PRINT*, ' ' 

DO 210 1=1,12 

DO 220 J=l,10 

DMAT( I , J) = MAT( I , J) 

220 CONTINUE 

210 CONTINUE 

DO 230 1=1,12 

DO 240 J = 1,10 

IF(DMAT( I , J). EQ. 1) THEN 
TEMP( I , J) = '*' 

ELSE 

TEMP( I , J) = '. ' 

END IF 

240 CONTINUE 

230 CONTINUE 

DO 140 K = 1, 12 

WRITE(*, 145) (MAT(K, J) , J=l,10), (TEMP(K,J), J=l,10) 

140 CONTINUE 

145 FORMAT( 4X, 1013 , 6X, 10( A1 , 2X) ) 

DO 10 1=1, N 

READ(*,15) (PATT( I , S) , S=1,M) 

10 CONTINUE 
15 FORMAT( IX, 815) 

*************************************************************** 

* ASSIGN CONNECTION WEIGHTS * 

****************■)'(■********************************************** 



DO 20 J=l, N 

DO 30 1=1, N 

IF(I.EQ.J) THEN 
T( I , J) = 0 

ELSE 

SUM = 0 
DO 35 S=1 , M 

SUM = SUM + (PATT(I,S)*PATT(J,S)) 
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CONTINUE 
T( I , J) = SUM 



ENDIF 

CONTINUE 

CONTINUE 



**************************************************************** 

* ITERATE UNTIL CONVERGENCE * 

**************************************************************** 



DO 40 K=1 , 9 

DO 50 J=l, N 
SUM 2 = 0 
DO 60 1=1, N 

SUM2 = SUM2 + (T(I,J)*U(K,I)) 
CONTINUE 

IF( SUM2. LT. 0) THEN 

U(K+1,J) = -1 

ELSE 

U(K+1,J) = 1 

ENDIF 
CONTINUE 
FLAG = 0 
DO 70 1=1, N 

IF(U(K,I).NE.U(K+1,I)) THEN 
FLAG = 1 

ENDIF 

CONTINUE 

IF(FLAG. EQ. 0) THEN 
COUNT = K 
GOTO 400 

ENDIF 

CONTINUE 



*************************************************************** 



PRINT SHAPES 

*************************************************************** 



DO 80 1=1, COUNT 
DO 90 J=l, N 

VEC(J) = U(I,J) 

CONTINUE 
PRINT*,' ' 

PRINT*, ' ' 

PRINT 260,1 

FORMAT( 'AFTER, THE',I2,'TH ITERATION S) , THE OUTPUT OF THE ' 
+ ' HOPFIELD NETWORK LOOKS LIKE '/ 

+ 'THE FOLLOWING FOR THE ' 

+ ' UNKNOWN INPUT PATTERN PRESENTED: ' ) 

PRINT*,’ ' 

PRINT*, ' ' 

CALL VECMAT( VEC , MAT) 

CALL CHARMAT(MAT,CMAT) 

CONTINUE 



************ ******sV****************************** ****** ********* 



82 



non 



* CLASSIFICATION * 

* ■> V -jV -jV ->V * * Vr * ■jV *>V Vc ?V -jV -jV Vc Vc Vr re rt Vr * V c ->V * ic -jV *V -jV Vc Vc ->V * Vc * -jV Vc ?V * Vc * -5V Vc Vc Vr Vc ■jV -jV -jV Vc Vc -jV V* Vc -jV Vc ?V ->V Vc Vc ?V Vc ‘V 



PRINT*,' ' 

PRINT*,' ' 

PRINT*, 'CLASSIFICATION OF THE UNKNOWN INPUT PATTERN: ' 

PRINT*, ' == ===== === ===== =^= : ======= == =========== ' 

PRINT 2 

2 FORMAT( ' AT THIS POINT. FURTHER ITERATIONS WON"T MAKE ANY ' 

&' CHANGE ON THE OUTPUT ' / 

&' OF THE NETWORK AND THE PATTERN SPECIFIED BY THE OUTPUT NODES ' 
&'IS THE'/ 

&' NET' ' S OUTPUT. THE TASK OF THE NET NOW IS TO CLASSIFY THE ' 

&' INPUT AS AN'/ 

&' ALREADY KNOWN PATTERN OR A NO MATCH WILL OCCUR. AFTER ' 

&' CLASSIFICATION, '/ 

&'THE OUTPUT PATTERN OF THE HOPFIELD NET MATCHES BEST THE PATTERN 
&' OF* ) 

DO 200 1=1, N 

CLASS ( I ) = U( COUNT, I) 

200 CONTINUE 

DO 180 S=1 , M 

DO 190 1=1, N 

DIFF(I,S) = PATT( I , S) - CLASS(I) 

IF(DIFF( I , S) . EQ. 0) THEN 
SCLASS = S 

ELSE 



GOTO 180 

ENDIF 

190 CONTINUE 

GOTO 170 
180 CONTINUE 
170 IF( SCLASS. EQ. 1)THEN 

PRINT* , 'DIGIT ZERO. ' 

ELSEIF( SCLASS. EQ. 2 ) THEN 
PRINT*, 'DIGIT ONE' 

ELSEIF( SCLASS. EQ. 3) THEN 
PRINT*, 'DIGIT TWO. ' 

ELSEIF( SCLASS. EQ. 4) THEN 

PRINT*, 'DIGIT THREE. ' 

ELSEIF( SCLASS. EQ. 5) THEN 
PRINT*, 'DIGIT FOUR. ' 

ELSEIF(SCLASS. EQ. 6) THEN 
PRINT*, 'DIGIT SIX. ' 

ELSEIF( SCLASS. EQ. 7) THEN 

PRINT*, 'BLOCK REPRESENTING THE POINT. ' 
ELSEIF( SCLASS. EQ. 8) THEN 
PRINT', 'DIGIT NINE. ' 

ELSE 

PRINT*, 'NO MATCH' 

ENDIF 
CLOSE (1) 

CLOSE (2) 

STOP 

END 



C 



t 
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c **************************************************************** 

c 

SUBROUTINE VE CMAT ( ARR , MAT ) 

DIMENSION ARR( 120) , MAT(12,10) 

K = 0 

DO 100 J=l, 10 

DO 110 1=1,12 

K = K + 1 
MAT(I,J) = ARR(K) 

110 CONTINUE 

100 CONTINUE 
RETURN 
END 
C 

c **************************************************************** 
c 

SUBROUTINE CHARMAT( MAT, CMAT) 

DIMENSION MAT( 12,10) ,CMAT( 12 , 10) 

CHARACTER* 1 TEMP( 12,10) 

DO 120 1=1,12 

DO 130 J=l,10 

CMAT( I , J) = MAT( I , J) 

130 CONTINUE 

120 CONTINUE 

DO 160 1=1,12 

DO 165 J = 1,10 

IF(CMAT( I , J) . EQ. 1) THEN 
TEMP( I , J) = ’*' 

ELSE 

TEMP( I , J) = ' 

END IF 

165 CONTINUE 
160 CONTINUE 

DO 166 I = 1, 12 

WRITE( 6 , 167 )(TEMP( I , J) ,J=1,10) 

167 FORMAT( 22X, 10(A1 ,2X) ) 

166 CONTINUE 
RETURN 
END 

C 

Q **************************************************************** 

C * HERE ARE THE 8 STORED EXEMPLAR PATTERNS USED IN THIS PROGRAM * 

C * FOR CONVENIENCE, I CHOOSED TO WRITE THEM IN COLUNMS WHERE * 

C * EACH ONE CORRESPONDS TO A STORED EXEMPLAR PATTERN. THE FIRST * 

C * COLUNM CORRESPONDS TO THE PATTERN OF A ZERO, THE SECOND OF A * 



c 


* ONE, 


THE 


THIRD 


OF A 


TWO 


, THE 


FOURTH OF A THREE, THE 


FIFTH OF * 


c 


* A FOUR, THE SIXTH OF 


A 


SIX, THE SEVENTH OF A POINT, 


THE EIGHT* 


c 


* AND 


LAST 


OF A 


NINE. 








* 


c 


**************************************************************** 


C$DATA 
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APPENDIX B. PROGRAMING THE HAMMING NET WHEN USED AS 

AN OPTIMUM CLASSIFIER : 



Using Fortran as programming language, the previously described operation algo- 
rithm of the Hamming net when used as a classifier was implemented with the 
Mainframe, and used to run some simulations as described in the simulation paragraph 
of the Hamming net. 



**** THESIS RESEARCH 

**** HAMMING NET SIMULATION PROGRAM 






J# 






**** BY M. H. KHAIDAR 

j* j* j 1 ^ <^4^ j* j* 



j« j- j» «u «jt* j# ju j«j« j«j« j- j«j« j« j# j« j- ^ j* VcV? Vc 



* THIS PROGRAM WAS MADE TO IMPLEMENT THE ALGORITHM OPERATION OF* 

* THE HAMMING NET, WHEN IT IS USED AS A CLASSIFIER, PROVIDED IN* 

* THE CHAPTER ABOUT THIS NET. * 

* VARIABLE DECLARATION : * 

* N = NUMBER OF NODES IN EACH EXEMPLAR * 

* M = NUMBER OF STORED EXEMPLARS * 

* PATT( I , J) = THE ITH ELEMENT OF THE JTH STORED EXEMPLAR * 

* THETA = THE THRESHOLD IN EACH NODE * 

* W(I,J) = THE CONNECTION WEIGHT FROM INPUT I TO NODE J * 

* X(I) = THE ITH ELEMENT OF THE INPUT PATTERN TO THE NET * 

* U(J,T) = THE OUTPUT OF NODE J AT TIME T * 

* EPSILON = THE VALUE OF WEIGHTS (INHIBITORY) BETWEEN * 

* DIFFERENT OUTPUT NODES * 



Vc *<t\ f? Vf Vc Vc *4\ Vc Vc V* Vv VcVcVc V* #V /W> 



r* «*% kL J4 •>*+ (kf • 



k!^kL kLkU 



INTEGER PATT( 120 , 10) , U(10,ll), MAT(12,10), q 
INTEGER THETA, I, J, K, T, MAP(12,10) 

REAL RESLT, EPSILON, W(120,10), SUM, SUM2, X(120), ARR(120) 
REAL CMAT( 12,10) 

CHARACTER* 1 TEMP( 12,10) 

PRINTS'',' HAMMING NETWORK IMPLEMENTATION* 

PRINT*,' = — ====== == ' 

EPSILON = 0. 08 



******** THETA (J) — N/2 — 120/2 = 60 ************************** 



THETA =60 



**************************************************************** 
* ASSIGN CONNECTION WEIGHTS * 






M = 10 
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N = 120 
DO 10 1=1, N 

READ(*,15) (PATT(I,J), J=l, M) 

10 CONTINUE 
15 F0RMAT( IX, 1015) 

DO 20 1=1, N 

DO 25 J=l, M 

W(I,J) = PATT( I , J)/2. 0 
25 CONTINUE 

20 CONTINUE 



Vf Vp Vf V? Vf Vc V? Vf Vf Vr V? V? V* Vf V? V? Vf V? V* *iV Vf Vv Vv V** Vf V? V? Vc V? V? V? V* Vf V? Vv “V Vf Vc V* V? Vv Vr Vc "#c V? Vf 

* INITIALIZATION WITH UNKNOWN INPUT PATTERN * 

j. j. j. j. j.. J. J.. Jji Jj J ; ^ Jj j ; j, j. y, j, jj, j. j. j, j, j, j„ j, j. j, ju j, j, j., ^ j, j, j, j, _i_. j,. j, j, y, j, j, j, j. 



OPEN( UNIT=1 , FILE=' INPUT* , STATUS=' OLD' ) 

DO 30 1=1, N 

READ(1,*) X(I) 

30 CONTINUE 
PRINT*,' ' 

PRINT*,' ' 

PRINT 1 

1 FORMAT('THE UNKNOWN INPUT PATTERN TO THE HAMMING NETWORK, ON THE’ 
& ' LEFT AS IM-7 

&' POSED ON THE NETWORK AND ON THE RIGHT IN A MUCH CLEAR' 

&' REPRESENTATION '/ 

&' WHERE EVERY ( * ) REPLACES A 1 AND EVERY ( . ) REPLACES A -1: ' ) 
CALL VECMAT(X,CMAT) 

CALL CHARMAT( MAT, CMAT, TEMP) 

PRINT*,' ' 

PRINT*,' ' 

DO 140 I =1, 12 

DO 141 J=l, 10 

MAP( I , J) = INT(CMAT( I , J) ) 

141 CONTINUE 
140 CONTINUE 

DO 142 I =1, 12 

WRITE ( 6 , 145) (MAP( I , J) , J=l, 10), (TEMP(I,J), J=l, 10) 

142 CONTINUE 

145 F0RMAT(4X, 1013 , 6X, 10( A1 , 2X) ) 

DO 40 J=l, M 
SUM = 0 
DO 50 1=1, N 

SUM = SUM + W(I,J)*X(I) 

50 CONTINUE 

SUM1 = SUM + THETA 
IF(SUMl.GT.O) THEN 

U(J,1) = SUM1 

ELSE 

U(J,1) = 0 

END IF 

40 CONTINUE 






ITERATE UNTIL CONVERGENCE 
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**************************************************************** 



80 



70 

60 



DO 60 T=l, 10 

DO 70 J=l, M 
SUM2 = 0 
DO 80 K=l, M 

IF(K.NE.J) THEN 

SUM2 = SUM2 + U(K,T) 

ENDIF 

CONTINUE 

RESLT = U( J,T) - SUM2*EPSIL0N 
IF(RESLT. GT. 0) THEN 

U( J,T+1) = RESLT 

ELSE 



U(J,T+1) = 0 

ENDIF 

CONTINUE 

CONTINUE 



**************************************************************** 



* THE OUTPUT OF THE HAMMING NETWORK 

**************************************************************** 



PRINT*, ' ' 

PRINT*,’ ' 

PRINT 2 

2 FORMATC'THE OUTPUT OF THE HAMMING NETWORK, WHERE EACH COLUNM ' 

8 1 REPRESENTS THE '/ 

8 ' OUTPUT NODE VALUES FOR THE CORRESPONDING CLASSES AT A CERTAIN ' 
8' NUMBER '/ 

8' OF ITERATIONS: ' ) 

PRINT*, ' ' 

PRINT*, ' ' 

**************************************************************** 

* ONLY TEN ITERATIONS ARE SUFFICIENT TO THE NET TO CONVERGE TO* 

* THE RIGHT ANSWER FOR OUR SIMULATIONS * 



PRINT*, 'NUMB. OF ITERATIONS= 1 ' , ' 2',' 3',' 4',' 5', 

+' 6',' 7',’ 8',' 9',' 10' 

PRINT*, ’ ' 

DO 90 J=l, M 
Q = J - 1 

WRITE(*,95) Q,(U(J,T), T=l,10) 

90 CONTINUE 

95 FORMAT( IX, ' FOR CLASS’ , 12 ,4X, 1015) 

PRINT*,' ' 

PRINT* , ' ' 

PRINT*, 'CLASSIFICATION OF THE UNKNOWN INPUT PATTERN: ' 

PRINT* , ’ = ==^— — : — z — L = ===== ===== — ~=^= ' 

PRINT 3 

3 FORMATC'THEN, THE DISTURBED UNKNOWN INPUT TO THE HAMMING NETWORK '/ 
8 'AFTER CONVERGENCE CORRESPONDS TO THE PATTERN STORED OF THE’) 

IF(U( 1, 10). GT. 0)THEN 

PRINT’-, 'CLASS ZERO. ' 
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ELSEIF(U(2, 10). GT. 0)THEN 
PRINT*, 'CLASS ONE. ' 
ELSEIF(U(3, 10). GT. 0)THEN 
PRINT*, 'CLASS TWO. ' 
ELSEIF(U(4, 10). GT. 0)THEN 

PRINT*, 'CLASS THREE. ' 
ELSEIF(U(5 ,10). GT. 0 ) THEN 
PRINT*, 'CLASS FOUR. ’ 
ELSEIF(U(6, 10). GT. 0)THEN 
PRINT*, 'CLASS FIVE. ’ 
ELSEIF(U( 7,10). GT. 0)THEN 
PRINT*, 'CLASS SIX. ' 
ELSEIF(U(8, 10). GT. 0 ) THEN 

PRINT*, ' CLASS SEVEN. ' 
ELSEIF(U( 9,10). GT. 0)THEN 

PRINT*, 'CLASS EIGHT. ’ 
ELSEIF(U( 10,10). GT. 0 ) THEN 
PRINT*, 'CLASS NINE. ’ 

ENDIF 

STOP 

END 






SUBROUTINE VECMAT(ARR.CMAT) 
DIMENSION ARR(120), CMAT(12,10) 
K = 0 

DO 100 J=l, 10 

DO 110 1=1,12 

K = K + 1 

CMAT( I , J) = ARR(K) 
110 CONTINUE 

100 CONTINUE 
RETURN 
END 



SUBROUTINE CHARMAT( MAT, CMAT, TEMP) 
DIMENSION MAT( 12,10), CMAT( 12,10) 
CHARACTER* 1 TEMP(12,10) 

DO 150 J=l, 10 

DO 160 1=1, 12 

MAT( I , J) = CMAT( I , J) 
160 CONTINUE 

150 CONTINUE 

DO 170 I =1, 12 

DO 180 J=l, 10 

IF(MAT( I , J). EQ. 1) THEN 
TEMP( I , J) = '*' 

ELSE 

TEMP( I , J) = '. ' 

ENDIF 

180 CONTINUE 

170 CONTINUE 
RETURN 
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END 



Vr***VfVf**VcV« , *Vf**V« , Vf****VfVf*****VfVc**Vf********Vf^Vf*Vf*^V*V?******V< ,, 5VVc , :V^VTV'>VVfVf 

* HERE ARE THE 10 STORED EXEMPLARS USED IN THE IMPLEMENTATION * 

* OF HAMMING NET. FOR CONVENIENCE, I CHOOSED TO WRITE THEM IN * 

* A MATRIX OF 10 COLUNMS AND 120 ROWS, WHERE EVERY COLUNM 

* CORRESPONDS TO AN EXEMPLAR OF 120 ELEMENTS REPRESENTING THE * 

* 12 BY 10 REPRESENTATION OF THE PATTERN AS SHOWN IN THE STUDY * 

* OF THE HAMMING NET. THE FIRST VECTOR COLUNM CORRESPONDS TO 

* THE PATTERN OF DIGIT 0, THE SECOND OF 1, THE THIRD OF 2, THE * 

* FOURTH OF 3, THE FIFTH OF 4, THE SIXTH OF 5 , THE SEVENTH OF * 

* 6, THE EIGHT OF 7 , THE NINTH OF 8 AND THE LAST OF 9. * 

* Vc ■> V Vc * * ‘V -k Vc * : V * iV -jV sV Vc 'V ■> Wr ■sV -k Vc Vr Vc Vr ->V * Vc * * * * * ■> V Vc Vc Vc ->V * Vc Vc Vc ■>> * ' V * * Vc Vc Vc -k tV *V * * * * * * Vc 



C$DATA 
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APPENDIX C. ART AND OPERATION OF THE CARPENTER / 

GROSSBERG NET : 



The following is a description of the ART net operation according to Carpenter / 
Grossberg [Ref. 12]. A cycle that traces the real time dynamics of ART network in re- 
sponse to arbitrary sequences of binary input patterns is depicted in Figure 42. 




In Figure 42a, an unknown input pattern I is presented to the net. Pattern I is then 
transformed into a pattern X of activation across the nodes. In other words, the input 
pattern I generates a short term memory (STM) activity pattern X across a field of 
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feature detectors F, . Grossberg sees short term memory (STM) as a way of keeping 
patterns active after the original input pattern has vanished. A short term memory is a 
persistent activity pattern in a set of neurons, maintained by nonlinear feedback system. 

The input pattern I also activates an orienting subsystem A, but pattern X at F, in- 
hibits A before it can generate an output signal. On the other hand, the pattern X of 
STM activities across F,, elicits an output pattern S of output signals from F v When a 
signal from a node in F, is carried along a pathway (the bottom-up adaptive filter) to 
F 2 , the signal is multiplied or gated by the pathway's long term memory (LTM) trace. 
The LTM-gated signal (i.e., signal times LTM trace), not the signal alone, reaches the 
target node. Each target node sums up of all of its LTM-gated signals, which results in 
pattern S generating a pattern T of LTM-gated and summed input signals to F 2 as shown 
in Figure 42a. The transformation from S to T is called an adaptive filter. The input 
pattern T to F 2 is quickly transformed by interactions among the nodes of F 2 . The re- 
sulting pattern of activation across F 2 is a new pattern Y. This new pattern, rather than 
the input pattern T, is stored in STM by F 2 . As soon as the bottom-up STM transfor- 
mation X -* L" is completed, the STM activities Y in F 2 elicit a top-down excitatory signal 
pattern U back to F, (Figure 42b). Only sufficiently large STM activities in Y elicit 
signals in U along the feedback pathways F, -» F,. As before, the top-down signals U are 
also gated by LTM traces and the LTM-gated signals are summed at F, nodes. Then, the 
pattern U of output signals from F 2 generates a pattern V of LTM-gated and summed 
input signals to F,. The transformation from U to V is thus also an adaptive filter. The 
pattern V is called a top-down template, or learned expectation. 

Two sources of input now perturb F,. the bottom-up input pattern I which generated 
the original activity pattern X and the top-down template pattern V that resulted from 
activating X. The amount by which activity in X is attenuated to generate X’ depends 
upon how much of the input pattern I is encoded within the template pattern V. In 
particular, F, acts to match V against I. Now, we will discuss how a match or mismatch 
of I and V at F, regulates the course of learning in response to the pattern I. 

When a mismatch attenuates STM activity across F„ the total size of the inhibitor}' 
signal from F, to A is also attenuated. If the attenuation is sufficiently great, inhibition 
from F, to A can no longer prevent the arousal source A from firing. Figure 42c shows 
how disinhibition of A can result in the release of an arousal burst to F 2 which equally, 
or nonspecificallv, induces selective and enduring inhibition of active population of F 2 . 

In Figure 42c, inhibition of Y leads to removal of the top-down template V, and 
thereby terminates the mismatch between I and V. Input pattern I can thus reinstate the 
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original activity pattern X across F,, which again generates the output pattern S from 
F i and the input pattern T to F 2 . Due to the enduring inhibition at F 2 , the input pattern 
T can no longer activate the original pattern V at F 2 . A new pattern }" is thus generated 
at F 2 by I (Figure 42d). 

The new activity pattern Y' reads out a new top-down template pattern V. If a 
mismatch again occurs at F„ the orienting subsystem is again engaged, thereby leading 
to another arousal-mediated reset of STM at F 2 . In this way, a rapid series of STM 
matching and reset events may occur. Such an STM matching and reset series controls 
the system's search of LTM by sequentially engaging the novelty-sensitive orienting 
subsystem. Although STM is reset sequentially in time via this mismatch mediated, 
self-terminating LTM search process, the mechanisms which control the LTM search 
are all parallel network interactions rather than serial algorithms. Such a parallel search 
scheme continuously adjusts itself to the system's evolving LTM codes. In general, the 
spatial configuration of LTM codes depends upon both the system's initial configuration 
and its unique learning history, and hence cannot be predicted a priori by a pre-wired 
search algorithm. Instead, the mismatch-mediated engagement of the orienting subsys- 
tem realizes the type of self-adjusting search. 

The mismatch-mediated search of LTM ends when an STM pattern across F, 
reads-out a top-down template (V) which matches I to the degree of accuracy required 
by the level of attentional vigilance, or which has not yet undergone any prior learning. 
In this case, a new recognition category is then established and a new bottom-up code 
and new top-down template are learned [Ref. 6], 
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APPENDIX D. PROGRAMING THE CARPENTER / GROSSBERG NET 



Using Fortran as programing language, the previousely described clustering algo- 
rithm of the Carpenter / Grossberg net was implemented with the Mainframe, and used 
to run some simulations as described in the simulation paragraph of the net. 



**** THESIS RESEARCH **** 

**** CARPENTER / GROSSBERG NET SIMULATION PROGRAM **** 

**** BY M. H. KHAIDAR **** 

Vr^V*VrVr*^V**Vr*Vr*Vc’^rVrVr^V*VrVr^V*^VVcVfVrVf**Vr**Vc**^V**^V**Vr*** , 5V , 5V'5V'>V , 3V'5V-3V - }V'>V , 3V , jV'jV - jV’5V-3V , jV'5 , c’’5V 



* 


W(I.J) 


* 

* 


COUNT 


* 

V? 

Vr 


B( I , J) 


Vc 

Vc 


RO 


* 

* 

* 


PATT( I 


Vc 


X(I) 


* 


JMAX 


Vc 


U(J,T) 




*********’ 



THIS PROGRAM WAS MADE TO IMPLEMENT THE ALGORITHM OPERATTION 
OF THE CARPENTER / GROSSBERG NET, WHEN IT IS USED AS A CLASS 
IFIER, PROVIDED IN THE CHAPTER FOR THIS NETWORK. 

VARIABLE DECLARATION : 

THE TOP DOWN CONNECTION WEIGHT BETWEEN INPUT 
NODE I AND OUTPUT NODE J 

THE NUMBER OF PATTERNS STORED IN THE MEMORY OF 
THE NET AT A CERTAIN TIME T (THIS NUMBER IS VAR-* 
I ANT IN TIME) * 

THE BOTTOM UP CONNECTION WEIGHT BETWEEN INPUT 
NODE I AND OUTPUT NODE J 

THE VIGILANCE THRESHOLD WHICH INDICATES HOW CLO-* 
SE AN INPUT MUST BE TO A STORED EXEMPLAR TO * 
MATCH * 



= THE VECTOR REPRESENTATION OF THE INPUT PATTERN * 
= THIS VARIABLE INDICATES THE CLASS THAT BEST * 
MATCHES THE INPUT PATTERN * 

U(J,T) = THE OUTPUT OF OUTPUT NODE J AT TIME T 

jl ju j# j 1 ^ j# j# j 1 # y# ^ j# ^ j 1 ^ 



INTEGER W( 64, 10) , ANS, COUNT, JMAX, J, I, K, PATT(64,10) 
INTEGER T, AMAT(8,8), MATRIX(8,8), TRUE, TIME 
REAL B( 64 , 10) , SUM, SUM1, SUM2 , SUM3 , SUM4 , RO, SUM5 , SUM6 
REAL EPSILON, RATIO, Y(64), RESLT, CMAT(8,8) 

REAL ARR( 64) , X(64), PMAT(8,8), BMAT(8,8), U(10,ll) 



* INITIALIZATION * 

^VVrVrVr^VVr*Vc**VcVc*VrVc**VrVr**Vc*VrVrVr*VrVc*Vr*7cVrVc^V*VcVrVc*^V*Vr^VVr***Vc***Vr-jV , jV'3V'5V'3V , 5V , 3V - 5V->V’5V 

PRINT*,' CARPENTER / GROSSBERG NETWORK IMPLEMENTATION' 

PRINT* , ' ===== ===== ' ===== ' 

DO 10 1=1, 64 

DO 20 J=l, 10 
W(I,J) = 1 
B( I , J) = 0. 125 



9S 



o o o o o 



20 CONTINUE 

10 CONTINUE 
COUNT = 1 



**************************************************************** 



* APPLY NEW INPUT * 

**************************************************************** 



25 

26 



250 

240 



400 



0PEN(UNIT=1 , FILE=' LETTER' , STATUS=' OLD* ) 

DO 25 1=1, 64 

READ( 1,26) PATT( 1,1) 

X(I) = PATT( 1,1) 

CONTINUE 
FORMAT( IX, 15) 

PRINT*, 1 ' 

PRINT* , ' ' 

PRINT*, 'THE FIRST INPUT PATTERN TO CARPENTER / GROSSBERG NET.: ' 
CALL VE CMAT ( X , CMAT ) 

PRINT*,' ' 

PRINT**,' ’ 



DO 240 1=1, 8 

DO 250 J=l, 8 

AMAT( I , J) = INT(CMAT( I , J) ) 
CONTINUE 
CONTINUE 

CALL CHARMAT( AMAT,PMAT) 

TRUE = 0 
JMAX = 1 
TIME = 0 
GOTO 800 

IF(TRUE. EQ. 4) GOTO 600 
PRINT*,' ' 

PRINT*,' ' 



PRINT* 

PRINT* 

PRINT* 

PRINT* 



PLEASE, ENTER YOUR CHOICE: ' 

(1) A NEW INPUT PATTERN' 

f 

(2) STOP' 



READ* , ANS 



IF(ANS. EQ. 1) THEN 

TRUE = TRUE + 1 
IF(TRUE. EQ. 1) THEN 
GOTO 1 

ELSEIF(TRUE. EQ. 2) THEN 
GOTO 2 

ELSEIF(TRUE. EQ. 3) THEN 
GOTO 3 



ELSE 



GOTO 4 

ENDIF 

ELSE 

GOTO 600 

ENDIF 

1 0PEN(UNIT=2 , FILE='E ' , STATUS= ' OLD ' ) 
DO 30 1=1, 8 

READ( 2,35) (MATRIX( I , J) , J=l,8) 
30 CONTINUE 
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35 FORMAT( IX, 815) 

GOTO 7 

2 0PEN(UNIT=3 , FILE='F', STATUS=' OLD ' ) 

DO 31 1=1, 8 

READ( 3 , 36) (MATRIX(I, J) , J=l,8) 

31 CONTINUE 

36 FORMAT( IX, 815) 

GOTO 7 

3 0PEN(UNIT=4, FILE=' FPRIME ' , STATUS='OLD' ) 

DO 32 1=1, 8 

READ(4,37) (MATRIX(I, J) , J=l,8) 

32 CONTINUE 

37 FORMAT( IX, 815) 

GOTO 7 

4 0PEN(UNIT=5, FILE=' FDPRIME ’ , STATUS=' OLD' ) 

DO 33 1=1, 8 

READ(5 ,38) (MATRIX(I, J) , J=l,8) 

33 CONTINUE 

38 FORMAT( IX, 815) 

7 DO 44 1=1, 8 

DO 45 J=l, 8 

BMAT(I,J) = REAL(MATRIX(I,J)) 

45 CONTINUE 

44 CONTINUE 

CALL MATVEC(BMAT,X) 

*VfVf**V:Vf*^VVfVnV****>V**^V;V*****^*Vf?V*V«V?V?VV»V*^V**VfVoV********VnVVf*Vf**** 

* COMPUTE MATCHING SCORES * 



DO 40 J=l, COUNT 
SUM = 0 
DO 50 1=1, 64 

SUM = SUM + B( I , J)*X( I) 

50 CONTINUE 

U(J,1) = SUM 
40 CONTINUE 

* * ‘V * * * * “V * * * * * * * * * * * * * * * * * * * * * * ‘V * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * 

* SELECT BEST MATCHING EXEMPLAR 

*** * * * * * *5 V -5 V * * * * * * * *** * ****** ** ***** ** iV?V?Wf ** ***** ******* ******** ** 

900 EPSILON = 0. 08 
DO 60 T=1 , 10 

DO 70 J=l, COUNT 
SUM1 = 0 

DO 80 K=1 , COUNT 

IF (K. NE.J) THEN 

SUM1 = SUM1 + U(K,T) 

ENDIF 

80 CONTINUE 

RESLT = U( J,T) - SUM1*EPSILQN 
IF(RESLT. GT. 0) THEN 
U(J,T+1) = RESLT 

ELSE 

U(J,T+1) = 0 



100 



ooooo ooooo 



ENDIF 

70 CONTINUE 

60 CONTINUE 

DO 90 J=l, COUNT 

IF(U( J,9). GT. 0) THEN 
JMAX = J 

ENDIF 

90 CONTINUE 

* VIGILANCE TEST 



RO = 0. 7 
SUM2 = 0 
DO 100 1=1,64 

SUM2 = SUM2 + X(I) 

100 CONTINUE 
SUM3 = 0 
DO 120 1=1, 64 

SUM3 = SUM3 + W(I,JMAX)*X(I) 
120 CONTINUE 

RATIO = SUM3 / SUM2 
IF( RATIO. GT. RO) THEN 
GOTO 200 

ELSE 

GOTO 300 

ENDIF 



* DISABLE BEST MATCHING EXAMPLAR * 



300 IF( TIME. NE. COUNT) THEN 
DO 46 J=l, COUNT 

TIME = TIME + 1 
IF(U( J, 1). NE. 0) THEN 

IF(J. NE. JMAX) THEN 
SUM5 = 0 
DO 47 1=1, 64 

SUM5 = SUM5 + B( I , J)*X( I) 

47 CONTINUE 

U(J,1) = SUM5 

ELSE 

U(J,1) = 0 

ENDIF 

ENDIF 

46 CONTINUE 

GOTO 900 

ENDIF 

PRINT* , ' BECAUSE , THE RATIO IS LESS THAN THE VIGILANCE THRESHOLD' 
PRINT*, 'THE INPUT PATTERN IS CONSIDERED TO BE DIFFERENT FROM ’ 
PRINT*, 1 ANY EXAMPLAR PATTERN STORED. THIS INPUT PATTERN IS 1 
PRINT*, ’THEN STORED WITH THE OTHERS AS A NEW EXAMPLAR. 1 
CALL VECMAT(X,CMAT) 

PRINT*, 1 ' 
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270 

260 



130 

140 

145 

48 

49 

200 



150 



157 

155 



PRINT*, 'THE UNKNOWN INPUT PATTERN TO CARPENTER/ GROSSBERG NET.: ' 
PRINT*,' ' 

DO 260 1=1, 8 

DO 270 J=l, 8 

AMAT(I.J) = INT(CMAT( I , J) ) 

CONTINUE 

CONTINUE 

CALL C HARMAT ( AMAT , PMAT ) 






* ADD THE NEW INPUT PATTERN TO THE MEMORY OF THE NET * 

JLJLJ«J^JLJ« J« J« J# J« ^ ^ ^ JUJ« J# JU J# J# J«» 



COUNT = COUNT + 1 

0PEN(UNIT=1, FILE=' LETTER' , STATUS= ' OLD ' ) 

DO 130 1=1, 64 

PATT( I , COUNT) = X(I) 

CONTINUE 
REWIND 1 
DO 140 1=1, 64 

WRITE( 1,145) (PATT(I , J) , J=l, COUNT) 
CONTINUE 
FORMAT( IX, 1015) 

SUM 6 = 0 
J = COUNT 
DO 48 1=1, 64 

SUM6 = SUM6 + W(I,J)*X(I) 

CONTINUE 
DO 49 1=1, 64 

B( I , J) = (W(I,J)*X(I)) / (0.5 + SUM6) 
W( I , J) = W( I , J)*X( I) 

CONTINUE 
TIME = 0 
GOTO 400 



* ADAPT BEST MATCHING EXAMPLAR * 

j# j-%*# ^ J« J# J« jl j# ^ j# y# j« jl ^ jl jl jl 



PRINTS 'BECAUSE, THE RATIO IS GREATER THAN THE VIGILANCE' 
PRINT*, 'THRESHOLD, THE INPUT PATTERN IS CONSIDERED ’ 
PRINT* , ' TO MATCH A STORED PATTERN WHICH IS UPDATED BY' 
PRINT*, ' PERFORMING A LOGICAL "AND" OPERATION BETWEEN ' 
PRINT*, ' ITS BITS AND THOSE OF THE INPUT PATTERN, AND' 
PRINT*, 'THE NEW UPDATED PATTERN WILL LOOK LIKE: ' 

DO 150 1=1, 64 

Y( I ) = PATT( I , JMAX)*X( I) 

CONTINUE 

CALL VECMAT( Y,CMAT) 

PRINT*,' ' 

DO 155 1=1, 8 

DO 157 J=l, 8 

AMAT( I , J) = INT(CMAT( I , J) ) 

CONTINUE 

CONTINUE 
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CALL CHARMAT ( AMAT , PMAT ) 






* THE UPDATED PATTERN IS PUT BACK INTO THE MEMORY OF THE NET * 






0PEN(UNIT=1 , FILE=' LETTER’ , STATUS=' OLD' ) 

DO 160 1=1, 64 

PATT( I , JMAX) = Y(I) 

160 CONTINUE 
REWIND 1 
DO 170 1=1, 64 

WRITE( 1,175) (PATT(I,J), J=l, COUNT) 

170 CONTINUE 

175 FORMAT( IX, 1015) 

800 SUM4 = 0 

DO 180 1=1, 64 

SUM4 = SUM4 + W( I , JMAX)*X( I) 

180 CONTINUE 

DO 190 1=1, 64 

B ( I , JMAX) = (W( I , JMAX)*X( I ) ) / (0.5 + SUM4) 
W( I , JMAX) = W( I , JMAX)*X( I) 

190 CONTINUE 
GOTO 400 
600 CLOSE (1) 

CLOSE (2) 

CLOSE (3) 

CLOSE (4) 

CLOSE (5) 

STOP 

END 






SUBROUTINE VE CMAT ( ARR , CMAT ) 

DIMENSION ARR( 64) , CMAT(8,8) 

K = 0 

DO 220 J=l, 8 

DO 230 1=1,8 

K = K + 1 

CMAT( I , J) = ARR(K) 

230 CONTINUE 

220 CONTINUE 
RETURN 
END 

* ■jV * -jV ■>> * * * * •jV -,V * * -jV •jV -jV * * * * * Vc -jV -jV -jV ->V * it it it it it * it it it it it it it * * it * it it * * * it it it it it * it it * it * * it * 



SUBROUTINE MATVEC( CMAT, ARR) 
DIMENSION ARR( 64) , CMAT(8,8) 

K = 0 

DO 310 J=l, 8 

DO 320 1=1, 8 
K= K + 1 

ARR(K) = CMAT( I , J) 
320 CONTINUE 
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310 CONTINUE 
RETURN 
END 
C 

C **»v***************************************»v********************* 
C 

SUBROUTINE CHARMAT( MAT , CMAT ) 

DIMENSION MAT( 8,8) ,CMAT( 8 ,8) 

CHARACTER*1 TEMP(8,8) 

DO 330 1=1,8 

DO 340 J=1 , 8 

CMAT(I.J) = MAT(I,J) 

340 CONTINUE 

330 CONTINUE 

DO 650 1=1,8 

DO 660 J = 1,8 

IF(CMAT(I , J) . EQ. 1) THEN 
TEMP( I , J) = '*' 

ELSE 

TEMP( I , J) = ' ' 

ENDIF 

660 CONTINUE 

650 CONTINUE 

DO 370 I = 1, 8 

WRITE( 6 , 167)(TEMP( I , J) , J=1 ,8) 

167 F0RMAT(8X, 10A1) 

370 CONTINUE 
RETURN 
END 



For the first input pattern to the net, we have used the pattern of the letter "C" given 
below. The elements of the matrice representation take on 0 and 1 values. To make the 
pattern clearer, we have replaced even.' element of 0 value by a white pixel and elements 
of 1 value by black pixels. A compact representation of this pattern is shown to the right 
below : 



0 

0 

0 

0 

0 

0 

0 

0 



0 

1 

1 

1 

1 

1 

1 

0 



1 

0 

0 

0 

0 

0 

0 

1 



1 

0 

0 

0 

0 

0 

0 

1 



1 

0 

0 

0 

0 

0 

0 

1 



1 

0 

0 

0 

0 

0 

0 

1 



1 

0 

0 

0 

0 

0 

0 

1 



0 

1 

0 

0 

0 

0 

1 

0 



In a similar manner, the pattern representation of the letter "E" used in the simu- 
lation of the net is shown below. Where the left hand side pattern representation of "E" 
is the actual input to the net. 
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0 

0 

0 

0 

0 

0 

0 

0 



11111 
1 0 0 0 0 
1 0 0 0 0 
1110 0 
1 0 0 0 0 
1 0 0 0 0 
1 0 0 0 0 
11111 



0 0 
0 0 
0 0 
0 0 
0 0 
0 0 
0 0 
1 0 



The pattern of the letter "F" is represented as : 



0 1 
0 1 
0 1 
0 1 
0 1 
0 1 
0 1 
0 1 



1 1 
0 0 
0 0 
1 1 
0 0 
0 0 
0 0 
0 0 



1 1 
0 0 
0 0 
1 0 
0 0 
0 0 
0 0 
0 0 



1 0 
0 0 
0 0 
0 0 
0 0 
0 0 
0 0 
0 0 



The noise corrupted version of the pattern "F" is 



0 110 
0 10 0 
0 10 0 
0 111 
0 10 0 
0 10 1 
0 10 0 
0 10 0 



1110 
0 0 0 0 
0 0 0 0 
10 0 0 
0 0 0 0 
0 0 0 0 
0 0 0 0 
0 0 0 0 



A noisier pattern of the letter "F" is : 



0 111 
0 10 1 
0 10 0 
0 111 
0 10 0 
0 0 0 0 
0 10 0 
0 10 0 



1110 
0 0 0 0 
0 0 0 0 
10 0 0 
0 0 0 0 
0 0 0 0 
0 0 0 0 
0 0 0 0 
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APPENDIX E. THE PARAMETERS FOR THE MLSE NEURAL 

NETWORK 

From [Ref. 3], the MLSE cost function for the time- varying channel can be written 
as 



M 



m ,vr 



h . /({««)) = - Y} a i z i + ZZ ]°i S i-k a k 



(£- 1 ) 



/=i 



i= 1 k= 1 



where 5, <2* denotes the value of s at time the i' h observation, z„ is sampled. The coeffi- 
cients van, - with the time and in general 



,<0 _*.</> » _t : 

^ 5 i-A- ‘ ^ i 



and it follows that 



W'ik — s i-k ^ s k—i ~ ^ ki ‘ ^ ^ 

thus, the symmetry condition \ 'hich is sufficient for stability no longer holds. 

The MLSE cost function can be reformulated such that the synaptic intercon- 
nections are symmetric. Consider the quadratic term of Equation E-l, 



M XI 

<£-2) 

i=\k= 1 

Let a and /? be two integers between 1 and M and assume for the moment that a # /?. 
Then two of the terms in the summation given by Equation E-2, one for i = a, k = /? 
and the other for i = /?, k = a, are 

(a) p 0 ?) 

a i s *-p a p & a p s p-* a <x 

respectively. Thus, for indices a and /? the summation given by Equation E-2 contains 
the term 



„ c< a) o + n *<P> fl =-L „ n fp (3) i { WU JL a a iJf) . M IfF-ll 

^3 — p ®p ^ a ^3 -) ^3 y^a—p -y? — 3/ ' -a ^3 W/?— 3 ' ^i—p } N 



106 



The two terms on the right side of Equation E-3 are identical. Define the modified co- 
efficient 5 ',_* as 



* i-k = 



= -L 



(& + 45 : 



(k) ) 



(£- 4 ) 



clearly, the modified coefficients are symmetric independent of time in the sense that 

s 'i-j = s 'j-i 

When i = k, the modified coefficient becomes 



= — (s/i + s£) = s l 0 



as desired. Also, using the property given by 



= s -i 



the stationary channel case reduces to 



^ i—k i-k "T ^k—i) $i—k 



Therefore, the MLSE cost function can be written with symmetric 5 ','s in a general form 
suitable for either stationary or time-varying channels as 

m m u 

I 

/= 1 (=1 k=l 

where the s',.*'s are given by Equation E-4. Using the MLSE cost function for the 
time- varying channel, the parameters for the MLSE neural network are given by 

2z, - /, , -2 s‘,. k - + sS) - IV, k , a, - v,(r) 



+ 






l-k a k 



MM) = - 
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APPENDIX F. PROGRAMMING THE MLSE NEURAL NETWORK 



Using Fortran as programing language, the MLSE neural network, described in 
Chapter V, was implemented with the Mainframe, and used to run some simulations for 
different network parameters and different transmission channel conditions (stationary 
or time-varying channel). 



**************************************************************** 
**** THESIS RESEARCH **** 

**** SIMULATION PROGRAM OF THE MLSE NEURAL NETWORK **** 

**** BY M. H. KHAIDAR **** 

**************************************************************** 



* 



* 



* 



**************************************************************** 

* THIS PROGRAM WAS MADE TO IMPLEMENT AND SIMULATE THE MLSE NEU-* 

* RAL NETWORK FOR A STATIONARY OR TIME-VARYING CHANNEL. THE PR-* 

* OGRAM FIRST WILL ASK THE OPERATOR TO ENTER THE DATA NECESSARY* 

* TO FULLY DESRIBE THE PROBLEM. THEN, THE PROGRAM IS GOING TO * 

* ASK THE OPERATOR IF THE PROGRAM IS TO BE RUN FOR A STATIONARY* 

* CHANNEL OR A TIME-VARYING CHANNEL. AFTER THE CHOICE IS MADE * 

* THE COMPUTER IS GOING TO DISPLAY THE SIMULATION RESULTS. 

* VARIABLE DECLARATIONS : 

* IR( 2500) = DATA SEQUENCE OF 2500 BITS OUTPUT OF THE GGUD 
IMSL SUBROUTINE (= A(2600)) 

M = THE NUMBER OF NEURONS IN THE NEURAL NETWORK 

INPUT TO THE PROGRAM, THIS TIME IT IS 17. 

MITH, MITER, INDEX, XEND, IWK(17), IER, WK(290), TOL, H=ARE* 
THE DESCRIPTION PARAMETERS OF THE PROBLEM TO THE * 
DIFFERENTIAL EQUATIONS SOLVER DGEAR. * 

L = CHANNEL MEMORY IN UNITS OF T (L = 2 FOR THIS * 

SIMULATION), INPUT TO THE PROGRAM. * 

N = THE NUMBER OF DATA TRANSMITTED (= NR), INPUT TO * 

THE PROGRAM. * 

P = THE NUMBER OF DATA BITS SHIFTED INTO THE REGIST-* 

ERS AT ONCE, INPUT TO THE PROGRAM. * 

COUNT = THE NUMBER OF DATA BITS THAT DIFFER BETWEEN THE * 
TRANSMITTED DATA AND THE MLSE NEURAL NET ESTIMATED* 
DATA, OUTPUT OF THE PROGRAM. * 

NUMBER = THE NUMBER OF DATA THAT DIFFER BETWEEN THE MLSE * 
NEURAL NET ESTIMATED DATA BITS AND THE DIRECT MLSE* 
COST FUNCTION CALCULATED DATA BITS, OUTPUT OF THE * 
PROGRAM. * 

REG( 17 ) = DATA BITS IN THE 17 REGISTERS. * 

G = GAIN FACTOR OF THE NEURAL AMPLIFIERS. * 

VOUT( 17) = THE DATA BITS OUTPUT OF THE NEURAL AMPLIFIERS. * 

AOUT( 2500) = ALL THE VOUT(17) WILL BE COLLECTED TO FORM THE* 
HOLE ESTIMATED DATA BITS CORRESPONDING TO 2500 DA-* 
TA BITS TRANSMITTED. * 

IN( 2500) = THE 2500 DATA POINTS GENERATED BU GGUD SUBROUTI -* 
NE TO FORM THE SAMPLES DELTA(2500) USED TO DESCRI-* 
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* 

* 

* 

Vf 

* 

* 

* 

* 

* 

* 

* 

* 

* 

* 

* 

* 

* 

* 

* 

* 

* 

* 

* 

* 

iV 

* 

* 

* 

* 



* 



* 

* 

)V 



* 



* 



* 

* 



* 



BE THE TIME -VARYING CHANNEL. (BETWEEN 1 AND 21) 
DELTA( 2500) = TIME-VARYING CHANNEL COEFFICIENTS (RANGE BET-* 
-0. 1 TO 0. 1). 

R( 2500) = THE 2500 GAUSSIAN NOISE SAMPLES USED TO IMPLEME-* 

THE PRESENCE OF NOISE IN THE CHANNELS. OUTPUT OF * 
THE GGNML IMSL SUBROUTINE. * 

U( 17) = THE 17 SOLUTIONS OF THE DIFFERENTIAL EQUATIONS * 

SOLVER DGEAR AND INPUTS TO THE NEURAL AMPLIFIERS. * 
T = TIME. * 

PERIOD = BIT DURATION (INVERSE OF THE DATA RATE) 

MUL = A MULTIPLIER USED TO IMPLEMENT THE TIME -VARYING * 

CHANNEL (MUL IS VARYING TOO). * 

V( 17 ) = EQUALS MUL AT A CERTAIN REGISTER. * 

TNOT = TIME DURATION OF THE INTERSYMBOL INTERFERENCE. * 

NNOT = THE SINGLE SIDED SPECTRAL DENSITY OF THE ADDITI-* 

VE WHITE GAUSSIAN NOISE N(T). * 

GN( 2600) = GAUSSIAN NOISE SAMPLES GENERATED BY THE GGNML * 
SUBROUTINE. * 

GNREG(17)= GAUSSIAN NOISE SAMLES INTO THE 17 REGISTERS OF * 
THE NEURAL NETWORK. * 

Z( 17) = THE 17 OBSERVATIONS OF THE STATIONARY CHANNEL * 

CALCULATED AS DESCRIBED IN THE STUDY. * 

Y( 17) = THE 17 RECEIVED SAMPLES FOR A STATIONARY CHANNEL* 

CALCULATED AS DESCRIBED IN THE STUDY. * 

YPRIME( 2500) = EQUIVALENT TO Y(17) BUT THIS TIME WHEN * 

CALCULATING FOR THE MLSE COST FUNCTION. * 

ZPRIME( 2500) = EQUIVALENT TO Z(17), FOR THE MLSE COST 

FUNCTION. * 

MLSECF( 2500)= THE 2500 SAMPLES GENERATED BY DIRECT CALCULA-* 
TION OF THE MLSE COST FUNCTION. 

SNR = SIGNAL-TO -NOISE RATIO (INPUT TO THE PROGRAM). * 
VPRIME( 2500) = SAME AS V( 17) BUT NOW IT'S FOR THE MLSE COST* 
FUNCTION. * 

DR = DATA RATE (INPUT TO THE PROGRAM, MAXIMUM 2400). 

FCN = SUBROUTINE DESCRIBING THE M DIFFERENTIAL EQUATIONS* 

FCNJ = EXTRA SUBROUTINE BUT NECESSARY. * 

W( 17,17) = THE SYNAPTIC CONNECTION MATRIX FOR THE NETWORK * 
ACTUALLY IT'S AN M BY M MATRIX. 

TAU = TIME CONSTANT OF THE CIRCUIT. 

CURRENT( 17) = THE 17 INPUT CURRENTS TO THE 17 NEURONS OF * 
THE NEURAL NETWORK. 

VIN( 17) = THE 17 DATA BITS INTO THE 17 REGISTERS OF THE * 
NEURAL NETWORK. 



**************************************************************** 



INTEGER IR( 2500) , X, NR, A(2600), M, METH, MITER, INDEX, ANS 
INTEGER IWK( 17 ) , IER, L, N, P, COUNT, SUP, MIN, NUMBER, RESP 
INTEGER MAX, REG(17), G, C, D, Q, VOUT(17), AOUT(2500), IN(2500) 
INTEGER DOWN, UP 

REAL R( 2500) , U(17), WK(290), T, TOL, H, PERIOD, MUL, V( 17) 

REAL TNOT, NNOT, GN(2600), GNREG( 17) , SUM, FACTOR, DELTA(2500) 
REAL SUM1 , Z( 17) , F, S, SUM2 , YPRIME( 2500) , Y(17), SUM5, SUM6 
REAL SUM3 , ZPRIME( 2500) , SUM4, MLSECF( 2500) , SNR, SUM7 , SUM 8 
REAL SUM9 , VPRIME( 2500) , DR 
DOUBLE PRECISION DSEED 
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EXTERNAL FCN, FCNJ 

COMMON W( 17 , 17) , TAU, CURRENT(17), VIN(17) 






PRINT* , 1 NEURAL NETWORK AS A MLSE RECEIVER' 

PRINT* , ' = ======= ==-=_ ======== == ' 

PRINT*,' ' 

3000 PRINT*, ’PLEASE, ENTER YOUR CHOICE, DO YOU WANT TO : ' 
PRINT 1 

1 FORMAT( 8X , ' ( 1 ) ' , ' CONTINUE WITH THE PROGRAM'/ 

&8X, * ( 2 ) ' , ' ' , ' QUIT 1 ) 

RE AD*, RE SP 
IF(RESP. EQ. 1) THEN 
GOTO 2 

ELSE 

GOTO 3 



2 



END IF 
PRINT*,’ ' 
PRINT*, ' ' 
PRINT*, 'PLEASE, 
READ* , M 

PRINT*, 'PLEASE, 
READ*, DR 
PRINT*, 'PLEASE, 
READ* , L 

PRINT*, 'PLEASE, 
READ* ,TAU 
PRINT* , ' PLEASE , 
READ* , SNR 
PRINT*, 'PLEASE, 
READ*,N 

PRINT*, ’ PLEASE, 
READ* , P 
COUNT = 0 
DOWN = 1 
UP = M 



ENTER THE NUMBER OF NEURONS (M) : ' 

ENTER THE DATA RATE DESIRED (HZ) : ' 

ENTER THE CHANNEL MEMORY (L) : ' 

ENTER THE TIME CONSTANT (TAU) : ' 

ENTER THE SNR (IN DB) : ’ 

ENTER THE NUMBER OF SYMBOL TRANSMITTED (N) : 
ENTER THE NO. OF SHIFTED SYMBOLS IN THE REG. 



SUP = M 



MIN = 1 
NUMBER = 0 



MAX = M 



(P) 









* GENERATION OF THE N DATA SEQUENCE AND THE N GAUSSIAN NOISE 

* SAMPLES AND PUTTING THE FIRST M DATA BITS AND M NOISE SAMPLES* 

* INTO THE M REGISTERS OF THE NETWORK TO START THE PROCESSING. * 






PERIOD = 1 / DR 
TNOT = L * PERIOD 

NNOT = (3 * TNOT) / (4 * (10 ** (SNR / 10.0))) 
X = 2 
NR = N 

DSEED = 123457. 0D00 
CALL GGUD( DSEED, X, NR, IR) 

DO 10 I = 1, NR 

IF( IR( I). EQ. 2) THEN 
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A( I ) = -1 

ELSE 

AC I) = +1 

END IF 

10 CONTINUE 

DSEED = 123457. ODOO 
CALL GGNML( DSEED, NR, R) 

DO 20 I = 1, NR 
GN( I) = R( I ) 

20 CONTINUE 

DO 30 I = 1, M 

REG(I) = AC I) 

GNREG(I) = GN( I ) 

30 CONTINUE 
PRINT*,' ' 

PRINT*, ' ' 

PRINT*, ' YOUR DATA IS NOW ENTERED, THE NETWORK IS READY TO BE* 
PRINT* , ' SIMULATED. THIS PROGRAM CAN SIMULATE THE MLSE NEURAL' 
PRINT*, 'NETWORK IN TWO CONDITIONS OF TRANSMISSION CHANNEL : ’ 
PRINT* , ' ’ 

PRINT 5 

5 FORMAT ( 8X . ' ( 1) ' , ' ' , ' IN A STATIONARY CHANNEL' / 

&8X, ' (2) V ' , ' IN A TIME-VARYING CHANNEL' ) 

PRINT*,' ' 

PRINT* , ' PLEASE , ENTER YOUR CHOICE : ' 

RE AD*, AN S 
PRINT*,' ' 

PRINT*,' ' 

PRINT 12 

12 F0RMAT(12X,' PROCESSING IN PROGRESS PLEASE WAIT') 

PRINT*,' ’ 

PRINT*,’ ’ 

IF( ANS. EQ. 1) THEN 
GOTO 200 

ELSE 

GOTO 500 

END IF 



■>V ■jV * “V Vf ■sV ■sV * Vc *>V "jV * * * ■sV -sV Vc Vf * ‘V Vc Vr Vr * Vc Ve Vc ? V * * * V? 'k V? Vc ->V it ?V ? V V? ? V Vc Vc Vc } V V? -V * Vc •> V * 



* SIMULATION OF MLSE NEURAL NETWORK IN A STATIONARY CHANNEL * 



Vc Vr Vf ■? V Vr Vr/V Vr Vc rr Vr Vc V: V- Vr : V ? > Vr V- V: -sV Vc ?V Vr >V ■> V >V “V Vc Ve Vc i V Vr Vc /V “V -sV Vr “V Vc “V Vr : V ■sV Vr Vr : V Vc * Vr Vr V- ->V Vr “V “V “V “V Vc Vc 



200 DO 40 1=1, M 
SUM = 0 



DO 50 K = 1, M 

FACTOR = (I - K) * PERIOD 
SUM = SUM + REG(K) * F( FACTOR, TNOT) 
50 CONTINUE 

YC I ) = SUM + GNREG(I) 

40 CONTINUE 

DO 60 I = 1, M 
SUM1 = 0 
DO 70 K = 1, M 

FACTOR = (K - I) * PERIOD 
SUM1 = SUM1 + Y(K) * F( FACTOR, TNOT) 
70 CONTINUE 
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2(1) = SUM1 
60 CONTINUE 

DO 90 I = 1, M 

CURRENT(I) = 2 * 2(1) 

VIN(I) = REG(I) 

DO 100 K = 1, M 

FACTOR = (I - K) * PERIOD 
W( I ,K) = -2 * S( FACTOR, TNOT,NNOT) 
100 CONTINUE 

90 CONTINUE 
GOTO 1000 



* SIMULATION OF MLSE NEURAL NETWORK IN A TIME-VARYING CHANNEL * 



500 NB = 21 
NR = N 

DSEED = 123457. 0D0 
CALL GGUD( DSEED , NB , NR , IN) 

DO 310 I = 1, NR 

IF(IN( I). GE. 1. AND. IN( I) . LT. 11) THEN 
DELTA(I) = -INC I) / 100.0 
ELSEIF(IN(I).GE. 11. AND. IN(I). LT. 21) THEN 

DELTA(I) = ( IN(I) - 10 ) / 100.0 

ELSE 

DELTA(I) = 0 

ENDIF 

310 CONTINUE 
K = 1 

DO 320 I = DOWN , UP 

V(K) = DELTA(I) + 0. 9 
K = K + 1 
320 CONTINUE 

DOWN = DOWN + M 
UP = UP + M 
IF(UP. GE. N) UP = N 
DO 330 I = 1 , M 
SUM5 = 0 

DO 340 K = 1 , M 

FACTOR = (I - K) * PERIOD 
MUL = V(K) 

SUM5 = SUM5 + REG(K) * FPRIME ( FACTOR, TNOT, MUL) 
340 CONTINUE 

Y( I) = SUM5 + GNREG(I) 

330 CONTINUE 

DO 460 I = 1 , M 
SUM6 = 0 

DO 360 K = 1 , M 

FACTOR = (K - I) * PERIOD 
MUL = V(K) 

SUM6 = SUM6 + Y(K) * F PR I ME (FACTOR, TNOT, MUL) 
360 CONTINUE 

Z(I) = SUM6 
460 CONTINUE 

DO 370 I = 1 , M 
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CURRENT(I) = 2 * Z(I) 

VIN(I) = REG(I) 

DO 380 K = 1 , M 

FACTOR = (I - K) * PERIOD 
MUL = V(K) 

W(I,K) = -2 * SPRIME( FACTOR, TN0T,NN0T,MUL) 
380 CONTINUE 

370 CONTINUE 
1000 G = 10000 
T = 0. 0 

DO 80 I = 1, M 
U(I) = 0. 0 
80 CONTINUE 

T0L = 0. 00001 
H = 0. 000001 
MITER = 0 
METH = 1 
INDEX = 1 
XEND = 5 * TAU 






* AFTER DOING SOME CALCULATIONS NOW WE ARE READY TO CALL DGEAR * 

* THE DIFFERENTIAL EQUATIONS SOLVER TO SOLVE OUR M EQUATIONS. * 






CALL DGEAR ( M , FCN , FCNJ , T , H , U , XEND , TOL , METH .MITER , INDEX , I WK , WK , I ER ) 
IF(IER.GT. 128) THEN 
PRINT 13, IER 

13 FORMAT ( / , ’ WARNING !!!!!... IER = ’ , 15 ) 

GOTO 3 

ENDIF 






*THE OUTPUT OF DGEAR, SOLUTIONS TO OUR M DIFFERENTIAL EQUATIONS* 
*ARE PASSED THROUGH THE M NEURAL AMPLIFIERS TO GET THE MLSE NEU* 
*RAL NETWORK ESTIMATES OF THE M DATA BITS THAT ARE IN THE M REG* 
*ISTERS. * 
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DO 110 I = 1, M 

VOUT(I) = -TANH( G * U(I)) 
CONTINUE 

DO 111 I = 1 , M 

IF(REG( I). EQ. 0) VOUT(I) = 0 
CONTINUE 



Vc Vc /V m* * /V V* Vc * V? Vc Vc ^ ^ J ^i JL JL ^ ^ ^ 



*HERE THE DATA BITS OUTPUTS OF THE NEURAL AMPLIFIERS ARE COLLE-* 
*CTED SO THAT LATER WE ARE GOING TO HAVE THE HOLE 2500 ESTIMATE* 
*S OF THE 2500 DATA BITS TRANSMITTED. * 



Vc Vf VrVc Vc V? V? V? Vf Vf Vc Vc Vc V? j- j- j- ^ 



IF(MAX.EQ.M) THEN 
K = 1 

DO 115 J = 1 , M-l 

AOUT(K) = VOUT(J) 
K = K + 1 
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115 



116 



118 



117 



120 



165 



166 



130 



CONTINUE 

ELSE IF( MAX. NE. N) THEN 

DO 116 J = M-P , M-l 

AOUT(K) = VOUT(J) 

K = K + 1 
CONTINUE 

ELSEIF(MAX. EQ. N) THEN 
MPRIME = M - P 
DO 118 I = M , 1 , -1 

IF(REG(I). EQ. 0) THEN 
M = M - 1 

END IF 
CONTINUE 

IF(M. GT. MPRIME) THEN 

DO 117 J = MPRIME , M 

AOUT(K) = VOUT(J) 
K = K + 1 
CONTINUE 

ENDIF 

END IF 



Vc V: : V ic V? Vc •>> Vc •>> ->V -jV ?Y Vc Vc Vr Vc Vr •>> *5 V •> Y * ?Y Vc Vc if * ->Y * Vr ?Y * Vr * -jV * Vr 7V * Vc Vr Vr ->Y Vc * Vr * * * * * * * ‘V * * Vr * * Vr * * *>Y *5 V 



*HERE , THE ESTIMATES FROM NEURONS L+l THROUGH M-l ARE TAKEN AS * 
*VALID. THEN A COMPARISON BETWEEN THESE ESTIMATES AND THEIR COR* 
’^RESPONDING IN THE REGISTERS I.E. THE INPUT DATA BITS IS MADE * 
*AND THE RESULT IS RECORDED FOR LATER USE. 



* ic Vr * Vr Vc ■jV ‘/V 'k V: Vr V: V? * Vc it ‘V *>Y Vc if *>V ‘V *>V Vc ‘V V' *>V ' 



■ Vr Vr ->V Vc -jWr ‘V Vc-jV Vr * ->V Vr -jY ->Y -jV -?Y icific ~Y Vr Vr ->Y if “V tV Vc Vr Vc ->Y ->Y ‘V if if 



C = L + 1 
D = M - 1 
IF(MAX. EQ. M) THEN 

DO 120 I = 1, M 

IF(VIN(I). NE. VOUT(I)) THEN 
COUNT = COUNT + 1 

ENDIF 

CONTINUE 

ELSE IF( MAX. EQ. N) THEN 

DO 165 I = M, 1, -1 

IF(REG( I). EQ. 0) THEN 
M = M - 1 

ENDIF 
CONTINUE 
DO 166 I = C, M 

IF(VIN( I). NE. VOUT( I) ) THEN 
COUNT = COUNT + 1 

ENDIF 

CONTINUE 

ELSE 

DO 130 I = C, D 

IF(VIN(I). NE. VOUT(I)) THEN 
COUNT = COUNT + 1 

ENDIF 

CONTINUE 

ENDIF 
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c 

c 

c 

c 

c 

c 



140 

150 



155 



270 



160 



2000 



C 

C 



*HERE, THE DATA BITS IN THE REGISTERS FROM 1 TO P ARE NULLED * 
*THEN THE CONTENT OF THE SHIFT REGISTERS ARE SHIFTED TO THE RIG* 
*HT TILL THE CONTENT OF REGISTER 1 IS NONZERO. THEN, WE ARE FEE* 
*DING THE P EMPTY REGISTERS BY THE NEXT P DATA BITS OF THE TRAN* 
*SMITTED SEQUENCE FOR ANOTHER PROCESSING CYCLE. * 



DO 140 I = 1, P 
REG(I) = 0 
GNREG(I) = 0 
CONTINUE 
Q = 1 

IF(Q. LE.M) THEN 

IF((P+Q). LE. M) THEN 
REG(Q) = REG(P+Q) 

GNREG(Q) = GNREG(P+Q) 

ELSE 

REG(Q) = 0 
GNREG(Q) = 0 
ENDIF 
Q = Q + 1 
GOTO 150 
ENDIF 

IF(MAX. EQ. N) THEN 

PRINT*, 'THE NUMBER OF ERROR DATA BETWEEN THE TRANSMITTED’ 
PRINT*, ' BINARY SEQUENCE AND THE MLSE NEURAL NET OUTPUT' 
PRINT*, 'DATA IS : ' 

PRINT*,' ’ 

PRINT 270, COUNT 
FORMAT(2X, 'COUNT =',2X,I5) 

GOTO 2000 
ENDIF 

DO 160 I = 1, M 

IF(REG( I). EQ. 0) THEN 
MAX = MAX + 1 
IF(MAX. GT. N) THEN 
A( MAX) = 0 
GN(MAX) = 0 

ENDIF 

REG(I) = A(MAX) 

GNREG(I) = GN(MAX) 

ENDIF 

CONTINUE 

IF(MAX. GT.N) THEN 
MAX = N 

ENDIF 

IF(ANS.EQ. 1) THEN 
GOTO 200 

ELSE 

GOTO 500 

ENDIF 

IF(ANS.NE.l) THEN 
GOTO 300 

ENDIF 



**************************************************************** 
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* AFTER, THE MLSE NEURAL NETWORK HAS ESTIMATED THE N DATA BITS * 
^TRANSMITTED, WE ARE NOW GOING TO START CALCULATING THE DIRECT * 
*MLSE COST FUNCTION ESTIMATES OF THE N TRANSMITTED DATA BITS * 
*THEN, WE ARE GOING TO MAKE A COMPARISON BETWEEN THE TWO ESTIMA* 
*TES AND RECORD THE RESULT. * 

■jV ic V«V Vr Vr Vr it zk Vc “V Vr Vr Vr Vr Vr * “V -j V Vr Vr Vr Vc ‘V /V ix * “Wc * ->V Vr Vr Vc * -jV Vr Vr -jV -jV Vr -jV Vr Vr -W? ■jV Vc “V Vr * Vr Vr Vc Vr * Vr ‘V ->V VoV Vr 



700 DO 180 I = MIN, SUP 
SUM2 = 0 

DO 190 K = MIN, SUP 

FACTOR = (I - K) * PERIOD 

SUM2 = SUM2 + A(K) * F( FACTOR, TNOT) 

190 CONTINUE 

YPRIME(I) = SUM2 + GN(I) 

180 CONTINUE 

DO 230 I = MIN, SUP 
SUM3 = 0 

DO 240 K = MIN, SUP 

FACTOR = (K - I) * PERIOD 

SUM3 = SUM3 + YPRIME(K) * F( FACTOR, TNOT) 

240 CONTINUE 

ZPRIME(I) = SUM3 
230 CONTINUE 

DO 250 I = MIN, SUP 
SUM4 = 0 

DO 260 K= MIN, SUP 

FACTOR = (I - K) * PERIOD 

SUM4 = SUM4 + A ( I ) * S( FACTOR, TNOT, NNOT) * A(K) 
260 CONTINUE 

MLSECF(I) = 2 * A(I) * ZPRIME(I) - SUM4 
250 CONTINUE 
GOTO 350 
300 J = 1 

DO 470 I = MIN , SUP 

VPRIME(I) = V(J) 

J = J + 1 

470 CONTINUE 

DO 390 I = MIN , SUP 
SUM7 = 0 

DO 410 K = MIN , SUP 

FACTOR = (I - K) * PERIOD 
MUL = VPRIME(K) 

SUM 7 = SUM7 + A(K) * FPR I ME (FACTOR, TNOT, MUL) 

410 CONTINUE 

YPRIME(I) = SUM7 + GN(I) 

390 CONTINUE 

DO 420 I = MIN , SUP 
SUM8 = 0 

DO 430 K = MIN , SUP 

FACTOR = (K - I) * PERIOD 
MUL = VPRIME(K) 

SUM8 = SUM8 + YPRIME(K) * FPRIME( FACTOR, TNOT, MUL) 
430 CONTINUE 

ZPRIME(I) = SUM8 
420 CONTINUE 

DO 440 I = MIN , SUP 
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SUM9 = o 

DO 450 K = MIN , SUP 

FACTOR = (I - K) * PERIOD 
MUL = VPRIME(K) 

SUM9 = SUM9 + A( I )*SPRIME (FACTOR, TNOT, NNOT,MUL)*A(K) 
450 CONTINUE 

MLSECF(I) = 2 * A(I) * ZPRIME(I) - SUM9 
440 CONTINUE 

350 IF(SUP. EQ. N) GOTO 400 
MIN = MIN + M 
SUP = SUP + M 
IF( SUP. GT. N) THEN 
SUP = N 

ENDIF 

IF(ANS. EQ. 1) THEN 
GOTO 700 

ELSE 

GOTO 300 

ENDIF 

400 DO 280 I = I, N 

IF(AOUT(I). NE. MLSECF(I)) THEN 
NUMBER = NUMBER + 1 

ENDIF 

280 CONTINUE 

PRINT*, 'THE NUMBER OF ERROR DATA BETWEEN THE TRANSMITTED' 

PRINT*, 'BINARY SEQUENCE OF DATA AND DATA DIRECTLY GENERATED' 
PRINT*, 'BY THE MLSE COST FUNCTION : ' 

PRINT*,’ ' 

PRINT 290, NUMBER 
290 FORMAT(2X, 'NUMBER =',4X,I5) 

GOTO 3000 
3 STOP 
END 






*HERE IS THE SUBROUTINE CORRESONDING TO THE TRANSMISSION CHANNE* 
*L IMPULSE RESPONSE WHICH IS MODELED BY A FINITE RESPONSE SQUAR* 
*ED COSINE FUNCTION. THIS FUNCTION IS IMPLEMENTING THE STATIONA* 
*RY CHANNEL. * 






FUNCTION F( FACTOR, TNOT) 

REAL FACTOR 

REAL F, TNOT, PI, SUP 

PI = 3. 1415927 

SUP = TNOT / 2. 0 

IF(ABS(FACTOR). LE. SUP) THEN 

F = (C0S(PI *FACTOR / TNOT)) ** 2 

ELSE 

F = 0 

ENDIF 

RETURN 

END 



*HERE IS THE COMBINED RESPONSE OF THE CHANNEL AND MATCHED * 
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♦FILTER. THIS FUNCTION IS IMPLEMENTING THE STATIONARY CHANNEL. * 

* * ‘V * * ‘V * * * * * * 'V * * ** * * ■> V * * * * * * * ‘V * * * * * * * * * * * * -5? * * * * ‘V * * * * "V V: “V * * * * * * * * * * -V 



FUNCTION S( FACTOR, TNOT, NNOT) 

REAL FACTOR, S, TNOT, NNOT, PI 
PI = 3. 1415927 

IF(ABS(FACTOR). LE. TNOT) THEN 

S = (1 / (2 * NNOT)) * ((TNOT - ABS( FACTOR) ) 

& * (1 + 0.5 * COS( 2 * PI * FACTOR / TNOT)) 

& + (3 * TNOT / 4.0) * SIN (2 * PI * ABS( FACTOR) * TNOT)) 

ELSE 

S = 0 

END IF 

RETURN 

END 

**************************************************************** 
*HERE ARE THE SET OF DIFFERENTIAL EQUATIONS DESCRIBING THE DYNA* 
*MICS OF THE NEURAL NETWORK. 

**************************************************************** 



SUBROUTINE FCN(M,T,U,UPRIME) 

INTEGER M 

REAL U(M) , UPRIME(M) , T, SOM 

COMMON W( 17 , 17) , TAU, CURRENT(17), VIN(17) 

DO 210 I = 1, M 
SOM = 0 

DO 220 K = 1, M 

SOM = SOM + W( I ,K) * VIN(K) 

220 CONTINUE 

UPRIME(I) = SOM - (U(I) / TAU) + CURRENT(I) 

210 CONTINUE 
RETURN 
END 

**************************************************************** 



SUBROUTINE FCNJ(M,T,U,PD) 
INTEGER M 

REAL U(M) , PD(M,M) , T 

RETURN 

END 






♦SAME AS FOR THE FUNCTION F ONLY THIS TIME IS FOR THE TIME -VARY* 
*ING CHANNEL. * 
**************************************************************** 



FUNCTION F PR I ME (FACTOR, TNOT, MUL) 

REAL FACTOR 

REAL FPRIME , TNOT, PI, SUP, MUL 
PI = 3. 1415927 
SUP = TNOT / 2. 0 
IF(ABS(FACTOR). LE. SUP) THEN 

FPRIME = MUL * ((COS(PI *FACTOR / TNOT)) ** 2) 

ELSE 

FPRIME = 0 

ENDIF 
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RETURN 

END 



■3 V irk irk Vr Vr * * is * * Vc * Vr * * V? * k * * * * i> k * * * * k k * vc kk ?V * * -jV ~V Vc * * * k ?V * Vc * kk * * Vr * * k -jV VrsV * k * 

*SAME AS BEFORE (FOR THE FUNCTION S) ONLY THIS TIME IT IS FOR * 
*THE TIME -VARYING CHANNEL. * 

^A*****************^^****************************************** 



FUNCT ION S PRI ME ( FACTOR , TNOT , NNOT , MUL ) 

REAL FACTOR, SPRIME, TNOT, NNOT, PI, MUL 
PI = 3. 1415927 

IF(ABS(FACTOR). LE. TNOT) THEN 

SPRIME = ((MUL ** 2) / (2 * NNOT)) * ((TNOT - ABS( FACTOR) ) 
& * (1 + 0.5 * COS( 2 * PI * FACTOR / TNOT)) 

& + (3 * TNOT / 4.0) * SIN (2 * PI * ABS(FACTOR) * TNOT)) 

ELSE 

SPRIME = 0 

ENDIF 

RETURN 

END 
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c . 1 Computer implementation 

and simulation of some 
neural networks used in 
pattern recognition and 
classification. 



